Remote working may have become commonplace, but a team of earthbound doctors and scientists have taken the idea to new heights after using ‘holoportation’ technology to broadcast their likenesses to astronauts on the International Space Station (ISS).
The live video feed enabled two-way communications between the earthbound group – comprised of NASA flight surgeon Dr Josef Schmid, AEXA Aerospace CEO Fernando de la Pena Llaca, and their colleagues – and astronauts from ISS Expedition 65, which included eight astronauts spread across three Soyuz and SpaceX space launches.
Using a custom system developed by AEXA, which leverages Microsoft’s HoloLens Kinect camera to capture a scene in 3D, the astronauts wearing HoloLens augmented-reality glasses could see and talk with the earthbound scientists – even as the ISS orbited at 28,000 km/h at an altitude of 400km over their heads.
“This is a completely new manner of human communication across vast distances,” Schmid explained in announcing the successful technology test, which took place on 8 October last year.
The holoportation system will soon be expanded for two-way communication by projecting astronauts’ images back to the ground as well – enabling private medical conferences, psychiatric consultations, family meetings, and visits by VIPs to visit with astronauts on the ISS.
The need for better support of astronauts is more than theoretical, with the test happening just weeks after a planned space walk by astronaut Mark Vande Hei was delayed due to what officials described as “a minor medical issue”.
In the long term, the combination of holoportation with augmented reality (AR) technology has been flagged by NASA as a way of supporting spacebound crews on long-term interplanetary missions – allowing the space agency to provide specialised technical, engineering, medical, and other expertise as circumstances require.
This might, for example, see the designer of a critical technology or lab experiment interacting in real time with astronauts as they service the equipment in space.
With future plans including multi-site holoportation sessions and haptic technology, remote medical treatment, training, maintenance support and other interactions could be taken to a whole new level.
Wish you were here
The multi-camera holoportation technology has been in development since HoloLens was first announced in 2015, at which point ISS astronauts were already being seeded first-generation HoloLens units to experiment with augmented reality onboard the space station.
Microsoft updated HoloLens in 2019 – signing a deal to supply 120,000 of the units to the US Army for virtual battlefield training – and has refined the holoportation system to make it portable, cutting bandwidth consumption to just 30 to 50Mbps.
That’s low enough to be reliably transmitted across terrestrial wireless communications systems, paving the way for a range of use cases including aged-care provider Silver Chain’s ‘holographic doctor’ service, remote visits to a Spanish museum, and teaching medical students with a 3D patient simulator.
But its successful use on the ISS – whose high speed and 90-minute orbit make continuous data coverage challenging – presages its broader use in future long-term space missions.
Extending the system to more-distant spacecraft would pose bigger challenges, given that a Mars mission would have to compensate for transmission delays of up to 20 minutes each way.
“This is a brand-new way of human exploration,” said Schmid, “where our human entity is able to travel off the planet [but] our physical body is not there.”