A U.S. Office of Naval Research-sponsored lab showcases mixed, virtual and augmented reality technologies

for warfighters, researchers, government, industry and academia to collaborate. (U.S. Navy/John F. Williams)

Augmenting reality versus fixing reality

Now that we can integrate digital information into a user’s environment in real-time — the very definition of augmented reality, or AR — the possibilities seem endless. While the use of AR technology to overlay content onto the real world enriches a user’s perception of reality, it doesn’t replace it.

By John Heilprin

May 21, 2024

As our daily lives become ever more digital, we’re experiencing an even greater convergence of the real and virtual worlds with the advance of AR technology.

Augmented reality offers an astoundingly interactive, sensory experience that uses computer-generated perceptual information to enhance the real world. Software, apps, and hardware such as AR glasses overlay digital content onto real-life environments and objects to such an astonishing degree that people may not be equipped to handle the intense emotion of being instanteously immersed in virtual experiences such as big wall climbing, wingsuit flying, or swimming with sharks.

The 2023 GESDA Science Breakthrough Radar® tells us that connecting the virtual and physical worlds will require a new global computing infrastructure. Responsive 3D content tied to spatial information requires far more data and processing than the text, images and videos that make up today’s Internet. Over the next quarter century, breakthroughs in the processing power of AR devices means content no longer has to be served from the cloud, reducing infrastructure requirements again.

The Radar also cautions that AR will lead to a blurring of boundaries between physical and digital worlds, which could have profound impacts on human psychology and social relations. The reality-distorting effects of social media are a harbinger of what could come, and phenomena like disinformation, echo chambers and body dysmorphia caused by image filters could all be turbocharged by AR technology.

The personalization of AR experiences could ultimately lead to a breakdown in the shared reality that is essential for the smooth functioning of societies. Extensive psychology, neuroscience and social science research will be required to understand how AR will impact humanity and preempt potential pitfalls, the Radar forecasts, and VR simulations of future AR scenarios could provide crucial insights.

Augmented reality in its present form could provide a temporary respite from fixing reality. But what happens if it’s no longer possible to differentiate between what’s artificial and real? What if AR becomes so effective, it’s possible to walk outside without always knowing what’s part of the real environment?

AR technology can bring us the world we imagine. It also can track today’s real world and help us experience it in new ways. In Switzerland, for example, AR is already regularly used to overlay reality with digital geodata. SWISS Airlines launched an augmented reality app for one of its CS300 aircraft to mark a traditional winegrowers’ festival that occurs only once in a generation. More than half the Swiss population have already experienced AR without realizing it, according to the Swiss Augmented Reality Barometer, a study carried out by the University of Lucerne.

“It’s a bit similar to when people started to use the mobile phone. The consumer started to use more and more sophisticated applications, and it’s really become the main computing platform that everybody uses today,” Reto Grob, CEO of Augment IT, a leading Swiss software company in Bern and Zürich that specializes in mixed and augmented reality enterprise solutions, says in an interview. “You do almost everything with the phone, and your life is kind of wedded to it. We are now probably at the time where we are starting the first adoption of AR/VR for consumers in a broader scope. So if you look back in five, 10 years from now, maybe you could say today was ignition day.”

Grob says two of the best use applications for “spatial computing” glasses made by some of the market leaders, primarily Apple and Meta, is for training and onboarding employees, and for health care. “With augmented reality, you can train people on the spot where they work. Another big use case is the health sector,” he says. “This can be used for surgical planning or teaching how the body works. We work with a paraplegic center in  Switzerland, and it shows really the value of these devices. You can control everything with your eyes with these glasses. So people who cannot move anymore, they can start video calls with the family again. They can go and press the internet, or watch movies. And this adds so much value for them.”

Within the next several years, he anticipates, spatial computing glasses marketed at more affordable consumer prices will become a popular item to access live mass events, such as sporting matches and music concerts. Enjoying the comforts of home, people won’t have the same social interactions. But they will be able to enjoy the feeling of being in a front row seat, he says, accompanied by superb visual and audio access and “basically experiencing it as if you were there, being in very good places, VIP spots, the best spots in the stadium, which you usually can’t afford.”

Visual and hearing aids are a form of augmented reality that are firmly part of our lives today, and implant technologies could become the norm. In 2022, Silicon Valley company Mojo Vision announced the first major third-party consumer application to be tested on an AR contact lens with 30 times the pixel density of an iPhone. The contact lens, developed with Amazon’s support, can be used to shop more easily.

AR technology is poised to transform people’s lives particularly in health care and mobility, but more agreement is needed to set guardrails for ethical uses. The Council of Europe recently launched a strategic action plan on human rights and technologies in biomedicine. In March, London-based Cromwell Hospital reported becoming the first hospital in the United Kingdom and Europe to use an Apple Vision Pro mixed reality glasses as a logistics and organizational tool. The headset, run on AI-based software from augmented reality surgical software firm eXeX, helped surgeons perform microsurgical spine procedures.

“The software is seamless and has improved efficiency within the Complex Spine team,” said Syed Aftab, a consultant orthopaedic spinal surgeon.

 

Preserving cultural heritage and urban planning

In April, the U.S.-based AR Alliance announced that Facebook’s parent company Meta had joined as a founding member to help companies and organizations develop augmented reality hardware. “The promise of AR and its potential market are so vast that there is ample room for all our member companies to succeed together,” said Bharath Rajagopalan, an electrical engineer who serves as the alliance’s board chair.

The field of augmented reality application began in the 1990s, evolving with technological advances and developing information and communication technologies. AR adds information to a real environment, improving users’ experience with computer-generated graphics, sounds or touch feedback. AR is now used in everything from health care to education to transportation, but it has particular resonance for preserving cultural heritage.

Virtual and augmented reality technology can help people express themselves in ways beyond what’s possible now in the physical world. AR technology has also been used at UNESCO’s World Heritage Sites to emphasize their cultural, historical, scientific and other aspects of significance.

Sarah Kenderdine, a professor of digital museology at the Swiss Federal Institute of Technology in Lausanne (EPFL), has built breakthrough technologies in curation and cultural heritage. At EPFL, she built the Laboratory for Experimental Museology (eM+), exploring the convergence of imaging technologies, immersive visualization, digital aesthetics and cultural and scientific big data.

“We have been designing and building large-scale immersive virtual reality and augmented reality systems for about 20 years now,” Kenderdine told a GESDA summit panel on augmented and extended reality technologies. “They’re panoramic, hemispheric, augmented, panoptic and they offer us strategies for multi-sensory engagement and give us powerful ways to reformulate narrative.”

Japan’s Hachioji City combined AR technology with 3D city models to help citizens discuss opportunities for land redevelopment. The approach boosted public engagement, drawing people ranging from 20 to 60 years old, during the planned renovation of a public waste facility.

Participants used AR to visualize renovation ideas onto a map during a planning workshop that “received strong positive feedback and attracted a younger demographic than traditional planning events,” the Organization for Economic Cooperation and Development reported in February. “The promises of this new technology have inspired other cities in Japan to experiment with their public planning processes.”

 

The evolution of visualization methodology from the 18th century (Journal of Big Data)

 

‘Augmented reality will replace the mobile phone as our primary interface’

The first interactive AR technology came out of a parachuting accident over Dayton, Ohio, in the summer of 1992. Louis Rosenberg, an American mechanical engineer, was testing the Virtual Fixtures platform, the first interactive augmented-reality system that he developed at Wright-Patterson Air Force Base. He developed it to test how well “perceptual overlays” might improve human performance in manual tasks.

“The project began in 1991, when I pitched the effort as part of my doctoral research at Stanford University. By the time I finished—three years and multiple prototypes later—the system I had assembled filled half a room and used nearly a million dollars’ worth of hardware,” he recounted in IEEE Spectrum, published by the Institute of Electrical and Electronics Engineers. “And I had collected enough data from human testing to definitively show that augmenting a real workspace with virtual objects could significantly enhance user performance in precision tasks.”

Rosenberg also had worked in a NASA lab where researchers explored how to enable realistic, immersive simulated worlds. The term ‘virtual reality’ was coined in 1987 by Jaron Lanier, a computer scientist who sold the first VR headsets and wired gloves. Lanier’s work, in turn, was based on research by Ivan Sutherland, a computer scientist who pioneered computer graphics.

“One day at Wright-Patterson, where skydiving engineers tested experimental parachutes, a dummy rig fell to the ground when a parachute failed to open. Rosenberg examined the smashed sensors and cameras from the failed test with the dummy rig. “By some miracle, I was able to piece together two working units from the six that had plummeted to the ground,” he said. “And so, the first human testing of an interactive augmented-reality system was made possible by cameras that had literally fallen out of the sky and smashed into the earth.”

Today, Rosenberg says he believes that VR and AR technology, now referred to as the metaverse, will become an important part of most people’s lives by the end of the 2020s. “In fact, based on the recent surge of investment by major corporations into improving the technology,” he said, “I predict that by the early 2030s augmented reality will replace the mobile phone as our primary interface to digital content.”

The GESDA Radar forecasts the era of the open metaverse will begin about a quarter century from now.

The 10 most active countries in the field of AR in cultural heritage (Applied Sciences)

 

Where the science and diplomacy can take us

The 2023 GESDA Science Breakthrough Radar®, distilling the insights of 848 scientists from 73 countries, tells us people’s digital experiences have become a central part of their daily lives, thanks to the advent of personal computers, smartphones and social media, and that the real and virtual worlds are poised to become even more tightly enmeshed as augmented reality technology advances.

The findings in the 2023 Science Breakthrough Radar®

Based on the Radar, here’s where we stand in several important areas:

1.4 Augmented Reality

The technology is already finding applications in education, workplace training, and industry. In combination with data streams from the expanding Internet of Things, AR will create an intuitive new layer of information for workers — allowing them to easily and continuously check the status of industrial processes, for example, or providing navigation tools when operating in unfamiliar surroundings.

Radar, page 48

1.4.1 Augmented reality hardware

AR hardware developments have been spearheaded by advances in microelectronics, sensor technologies and vision equipment, which have also reduced the cost. With 5G availability on mobile phones, AR can be implemented in limited ways on portable devices without the need of expensive eyeglasses and Head-Mounted Displays (HMDs).

Anticipation in a nutshell

5-year horizon: Hardware begins to mature
10-year horizon: AR displays can be worn all day
25-year horizon: AR and VR begin to feel like reality

Radar, page 50

1.4.2 Augmented experiences

Early incarnations of AR will be concerned with two fundamental activities: learning about the world and interacting with other people. This will include tasks like overlaying instructions or translations on the real world, or participating in 3D video calls. Near real-time interaction is possible depending on the amount of processing required, and AR has moved from game-playing and applications requiring lower accuracy to precise engineering and medical fields.

Anticipation in a nutshell

5-year horizon: AR overlays for simple information
10-year horizon: Consumer-grade glasses and adaptive user interfaces
25-year horizon: Individually-tailored reality becomes the norm

Radar, page 51

1.4.3 AR platforms

Connecting the virtual and the physical worlds will require a new global computing infrastructure. Responsive 3D content tied to spatial information requires far more data and processing than the text, images and videos that make up today’s Internet. AR hardware, however, will not be capable of providing this alone. A massive expansion in cloud services will be necessary to render high-fidelity graphics at scale. Faster wireless technology, including 5G, and edge stations that can bring processing closer to the user, will be crucial for ensuring AR content is delivered with low latency.

Anticipation in a nutshell

5-year horizon: Early commercial AR applications emerge
10-year horizon: High-fidelity AR experiences transmitted wirelessly
25-year horizon: Era of the open metaverse begins

Radar, page 52

1.4.4 Human factors of AR

AR will lead to a blurring of the boundaries between the physical and digital worlds, which could have profound impacts on human psychology and social relations. The reality-distorting effects of social media are a harbinger of what could come, and phenomena like disinformation, echo chambers and body dysmorphia caused by image filters could all be turbocharged by AR technology. The personalization of AR experiences could ultimately lead to a breakdown in the shared reality that is essential for the smooth functioning of societies. Extensive psychology, neuroscience and social science research will be required to understand how AR will impact humanity and preempt potential pitfalls. VR simulations of future AR scenarios could provide crucial insights.

Anticipation in a nutshell

5-year horizon: Research maps out social implications of AR
10-year horizon: Government regulation begins
25-year horizon: Pervasive AR threatens shared reality

Radar, page 53

5.2.2 Educational sensing

We can now observe and examine learning practices using digital technologies. By gathering and analyzing anonymized data using computer-based vision technology, student-held devices and other tools, researchers are beginning to make sense of the best practices in teaching, and to understand what enables effective learning.

5-year horizon: Frameworks for sensing are established
10-year horizon: Sensing technology goes mainstream
25-year horizon: AI and wearables change the learning experience

Radar, page 221

5.2.4 Neuroscientific aspects of learning

Although investigations of neuroscience as applied to learning have yet to deliver significant tangible breakthroughs in educational philosophy or practice, there are reasons to continue efforts to understand how the brain functions when learning. A better grasp of the operations behind working memory, executive function, attention, cognitive flexibility, theory of mind, and inhibition, for example, would open up avenues for improving the efficiency and outcomes of education. The role of social factors is also an important subject of investigation here: has evolution equipped us to learn differently in groups as opposed to when we are alone?

5-year horizon: Progress in basic neuro-learning research
10-year horizon: Brain tech comes of age
25-year horizon: Augmented reality accelerates education

Radar, page 223