The strange future of AR and VR: Tentacles and time travel to digital doctors
It's been a busy few days: I've been a Victorian gentlewoman, I've been chased across London by death-ray-blasting Martian tripods, I've got lost in a fashion show filled with giant models, and I've helped an alien with their cantankerous time machine, disguised as a blue police box.
Right now, virtual reality (VR) and augmented reality (AR) experiences, like the one I was testing out, are all about this kind of high drama: slipping on a headset and stepping into a different world in a different body -- if only virtually.
There's a reason why AR and VR have such an impact on us. Humans are an extremely visual species, with around a third of the neurons in our brains devoted to vision. That's why something like a VR headset, which can completely alter what we think we can see, can have a profound impact -- especially once headphones and additional effects (like a breeze or some smells) are added. That vivid impact has led to a steady stream of predictions over the years that these technologies are about to become mainstream. VR and AR have been about five years away for at least two decades now.
SEE: Executive's guide to the business value of VR and AR (free ebook)
Sure, even the current crop of headsets, some with their own accompanying backpacks, are too heavy, too sweaty and too clunky. They can be prone to stuttering and delivering images that are fuzzy or blocky, or in the case of AR, creating virtual objects that don't behave as they should, like a chair floating in mid-air. Part of the fun is still in spotting the glitches in the VR matrix. Moving around in a VR world is confusing and can even make you dizzy, and using controllers with too many buttons when you can't see your hands means it's easy to get actions wrong.
But even with the current imperfect state of the art, it only takes a limited suspension of disbelief to almost believe you are wherever the VR designers want you to be, or that the digital objects overlaid on the AR display are as real as you are.
That the potential is now increasingly clear is what makes VR and AR so exciting. These technologies, although clunky, are at a similar stage to cinema in the early twentieth century, or TV in the 1930s: it's a new type of media -- even a new way of seeing the world -- gradually taking shape.
At the same time, VR and AR need to get past these 'fun' visions if they want to become ubiquitous technologies. Over time, virtual rollercoaster rides and space missions will not in themselves be enough to sustain VR and AR, even if they do provide a guaranteed source of colour for journalists writing about the often-dry subject of technology. It's perhaps fitting that if you try to describe one of these VR experiences, it's a bit like retelling a dream -- nonsensical and inexplicable to those who weren't there.
To make the leap from gimmick to transformative mainstream technology, VR and AR may need to get a little less exciting and a bit more useful.
AR and VR explained
AR + VR
VR uses a closed headset that places the user in a virtual world where they simply watch what is happening, like an immersive version of television, or actually move around (either in the virtual space and/or the real world) to interact with this environment and the people and things within it.
AR, by contrast, uses a headset or other screen through which the real world is still visible and overlays digital objects on it -- anything from a dragon, to a vase to some sales data -- to add context or appear to be part of it.
These technologies have existed in one form or another for decades, but were extremely expensive, cumbersome and limited.
In the last few years, thanks largely to the smartphone, the technology has advanced and the price fallen to the point where it's now viable for both business and consumer uses -- even if there are significant hurdles before it becomes mainstream.
SEE: The business guide to AR and VR: Everything you need to know
VR and AR are best considered as a continuum of technologies and devices at different levels of maturity across business and consumer markets.
Tech advisors Digi-Capital estimate that there will be 2.5 billion AR-capable devices -- smartphones and smart glasses -- by 2023, compared to roughly 900 million now, and the market will be worth $70 billion. That compares to around 30 million VR devices (including mobile, standalone, console and PC-based), which Digi-Capital predicts will create a market worth $10 to $15 billion in the same period.
App store revenues, ads and ecommerce are likely to be the biggest drivers of AR, with entertainment and gaming the main revenue sources for VR.
"Right now, AR is more impactful in the enterprise: you are seeing a lot of the near-term benefits of applying it to specific tasks like 'here is your checklist to do a job' or situation video. On the VR side, because a lot of the power comes from gaming, you see it a lot in gaming and entertainment," says Tuong Nguyen, principal analyst at tech research firm Gartner.
The outlook for VR
VR is easier to define as a market because the headset blocks off the real world to provide you with a digital alternative. The best-known uses are in gaming, which can require the headset being tethered to a powerful computer that provides the necessary processing power.
This is different to - say - console or PC gaming - in one important way, because it becomes a full-body experience. You aren't just aiming a controller and moving about on screen, you're using your feet to walk around and waving your arms, or turning around to spot something behind you. It's one thing navigating a figure on a screen across a narrow bridge using a controller; it's another thing entirely to try to do the same thing wearing a headset, even if you are 99% (OK, maybe 95%) sure you aren't really going to be hurt in the real world if you fall off in VR.
These games can take advantage of the way that VR can allow developers to create impossible worlds that could not be experienced in the real world except through fiction.
One example: Dr Who's Tardis can really be bigger on the inside in a VR game like The Edge of Time for the HTC Vive Cosmos headset, which I tried out when researching this feature. Playing the game, and stepping out of the Tardis onto a dark alien world, even if I knew I was, in reality, walking across a white-walled warehouse in London, was genuinely thrilling -- even if for me the fear of a) looking stupid and b) walking into a wall in the real world did take away a little of the the excitement.
The Edge of Time is a new VR experience developed by BBC Studios and Maze Theory.
VR is also increasingly being used in areas such as entertainment. A couple of years ago London's Tate Modern recreated the leaky, smoky garret of painter and sculptor Amedeo Modigliani in VR as part of its exhibition, and London is currently home to an immersive version of Jeff Wayne's musical version of HG Well's Victorian sci-fi The War of The Worlds. This features VR and AR alongside volumetric holograms and live actors.
The VR is used in two sections; first as people trying to escape Victorian London with the Martian tripods overhead, and then again when some of the survivors are trying to escape on a boat as the valiant HMS Thunder Child makes a doomed attempt to fight back against the invaders.
Audiences are thrown headfirst into the battle between Martians and humans.
It's fun and occasionally funny as you see your friends transformed into Victorian Londoners, as in this case you can see the other audience members. The VR isn't perfect and there's a certain amount of tongue-in-cheek (fitting as it's based on an 1970's rock opera), but it certainly gives you a buzz to see the Martian tripods striding overhead and letting loose with their heat rays.
A more rarified example of VR in action is on show at the National Theatre, which has been experimenting with the technology for some time. All Kinds of Limbo is a piece with music commissioned that takes the audience through reggae, grime, classical and calypso in a tour of music inspired by the influence of West Indian culture on the UK's music scene. It takes the form of a relatively traditional musical performance, albeit one that spins rapidly from era to era.
In this case, rather than being represented by human figures, the other members of the audience are represented by a single white line in space.
The National Theatre was drawn to VR by its potential for dramatic storytelling, says Toby Coffey, its head of digital development. Its use of VR is still evolving as writers and artists gain more experience with it. VR is often seen as quite solitary but, in the case of this production, couples or friends were happy to hold hands or even dance with the headsets on "which gives us a very interesting artistic proposition," Coffey says.
The National Theatre's All Kinds of Limbo is a VR musical tour through the influence of West Indian culture on the UK's music scene.
The business models are still being built, and it might be ten years before this technology is ready for the big theatres, but the National Theatre wants to ensure it's not left behind once the audiences start expecting it. That's not to say traditional forms of theatre will be replaced, but new technologies can augment it.
"In five years' time you might pay £50 for a ticket, but to say whether it was a film or a game, or a piece of theatre or a piece of VR might be less obvious because it's something that goes across all of those," says Coffey.
"This is not just for the entertainment industry, this is across everything, architecture, healthcare, education. I don't think in any way it's a technology looking for a home: it can be used as a gimmick and quite often it is, but this is like TV or games or radio -- this is like a whole new discipline," he says.
These theatrical offerings show off some of the exciting potential of VR, but it is more prosaic use cases that will drive it towards the mass market. One increasingly common area of potential for VR is training. By using headsets, companies can standardise training, and make it easier for staff to learn in their own place of work, cutting costs for travel and trainers and lost productivity.
Late last year Walmart said it was using more than 17,000 Oculus Go headsets in stores to train staff on new technology, customer service, and compliance, and is also using VR to test whether workers have the skills for middle-management. The company said that because VR makes training more of an experience it improves retention of new knowledge -- VR improved test scores 10 to 15%. Walmart has also developed an AR price-checking tool for customers to use.
Also in the US, Farmers Insurance is working on a VR training programme that uses an AI-powered virtual human to simulate workplace conversations to help employees practice interpersonal skills. It has already built a VR tool with 500 simulated damage combinations and scenarios to help staff practice home damage assessments.
VR is also being used in retail, alongside AR: Macy's is offering a VR experience across 90 stores where customers provide the dimensions of a room and the retailer lays out the virtual space; customers can then pick their pieces, design the room -- including wall colour, flooring and furniture -- and put on the VR goggles to see how the room looks.
Macy's said that in three pilot stores 'VR-influenced' furniture sales have increased by more than 60 percent versus non-VR furniture sales, while returns have dropped to less than 2 percent; the service also allows Macy's to offer a full range of furniture in a smaller space. The in-store service is coupled with an AR app that also allows customers to place virtual versions of Macy's furniture in their actual living spaces.
The outlook for AR
In contrast to VR, AR overlays digital features onto a view of the real world -- something that can be done in many ways.
"I like to think of it as similar to the internet, it's that next step that takes the internet into the real world," says Gartner's Nguyen.
The most obviously futuristic of these are smart glasses -- glasses that incorporate a display. However these are at an early stage for consumers and there are plenty of other ways to do AR. An AR display could be a smart mirror that shows how you would look wearing an item of clothing. Many of us already carry an AR device, as most high-end smartphones are now capable of adding digital elements to what we see on-screen, whether that's using an app like Snapchat or a game like Pokémon Go.
There are a few high-profile AR apps already. In 2016 Pokémon Go placed virtual creatures all over the world for players to view through their smartphone screen, as if they were standing in front of them in the real world. The game was a sensation and has been downloaded one billion times, generating billions in revenue for the developers and showing that AR apps could be a huge success. However, subsequent AR games have failed to generate similar levels of excitement.
Other apps, like the Ikea Place iOS app, allow users to place virtual objects -- in this case, furniture from the Swedish retailer -- around a room to see how they would look.
For consumers in particular, smartphone-based AR will drive the market in the short to medium term. Apple (ARKit) and Google (ARCore) have built AR capabilities into iOS and Android respectively, which means that newer and high-end handsets will already have AR built in.
But that doesn't mean that the technology is being actively used: most smartphone owners will be unaware it's there, and that's unlikely to change without the arrival of a new killer app. What's likely is that over time more AR features will be built into existing apps like Google Maps, which now offers AR directions in the forms of arrows overlain on the map on the smartphone screen.
As a result, what's most likely to drive the AR market in the short term is enterprise usage. Using headsets to help guide staff through complicated tasks like fixing a particular piece of machinery or some other checklist is already a standard use case for AR headsets, which are used across a number of industries.
In November last year Microsoft won a two-year contract worth $480m with the US government to bring AR headsets to US soldiers, using HoloLens for training and on the battlefield with the addition of thermal imaging. Early versions allow troops to see other soldiers or satellite maps of terrain inside the headset. This contract could result in the US Army buying around 100,000 mixed-reality headsets from Microsoft -- potentially making the US Army the biggest user of the headsets over time.
SEE: Using AR and VR to train surgeons
Another example is a research project at Imperial College Healthcare Trust in London, which is using Microsoft's mixed-reality headset to plan plastic surgery, identifying which tissue and veins can be used in reconstructive operations. Indeed, healthcare is becoming a rich area for AR and VR experimentation, with everything from VR projects to tackling social anxiety to AR for training doctors. While these projects are currently small, they show potential for growth.
What's holding back AR and VR?
VR and AR have been about five years from the mainstream for the last two decades. While the technology is vastly superior to 20 years ago, there are still significant limitations. VR headsets are still too heavy (many come with weighty backpacks), which makes them hard to use over long periods, and too expensive and too complicated for the casual user. Even worse, to get the best graphics you're still required to be tethered to a PC. While individual VR experiences can be remarkable, they are still limited.
"An AR/VR dragon or portal is impressive the first time you see it, but gets old pretty quickly. The next stage of AR/VR must deliver against critical use cases, with features in critical apps that we use all day, every day," says Digi-Capital.
There's still a lack of broad content for headsets, let alone a killer app, which will also limit the market. With so many different pieces of hardware competing it's hard to know which to buy or develop for. There's also the issue of trust; one of the VR experiences I tried out glitched so that the virtual room no longer matched up with the real room, which meant I spent a lot of time walking to walls.
AR has different problems. While the installed base is already massive thanks to smartphones, there is lack of awareness and, beyond Pokémon Go, chasing a very limited set of apps that make good use of the technology. AR headsets will be mostly an enterprise play for the next few years, and will remain bulky and unattractive to most consumers.
The outlook for AR and VR
The market for VR is likely to be much more limited than that for AR. While Facebook's Mark Zuckerberg may want one billion people trying out VR, the idea of millions inhabiting virtual worlds and ignoring the real world around them is worrying but frankly unlikely. However, Facebook's recent acquisition of CTRL-Labs suggests that Zuckerberg is not giving up any time soon. Its technology could allow us to give up on using cumbersome controllers in VR, making the experience more natural.
Right now, AR still lacks both a killer app and killer piece of hardware to kick-start mass adoption. Rumours persist that Apple is working on some type of smart glasses, which might at first be tethered to an iPhone. Certainly most smartphone makers, who are struggling to get users to upgrade their handsets, are hoping that there is a 'next big thing' they can persuade us to buy. If smart glasses can be made that aren't a rerun of Google Glass, then AR usage could leap.
"I wear glasses on a regular basis. If you can give me some of that functionality in the device -- my glasses -- that I already use, that's phenomenal, so I don't have to pull out my phone or pull out my tablet to get this additional information," says Gartner's Nguyen.
If AR becomes a success it will also raise a fresh set of questions about privacy. Allowing corporations (or even governments) to overlay data or objects in our field of vision will literally change the way we see the world. Giving these companies access to everything we see is a step beyond what even the most committed social media fan may be willing to do. And don't forget that advertising is likely to be a big driver of AR revenues: if you thought being chased around the internet by ads for a pair of shoes you've already bought was bad, imagine them superimposed on people walking down the street in front of you.
AR and VR are incredibly intimate experiences, so we need to be able to trust the technology and the motives of the companies developing it. Big tech has a lot to do to persuade us that they are worthy of that trust.
While the 'always-five-years-from-mainstream' reputation of VR and AR would suggest we have a while to confront these challenges, a view needs to be taken sooner rather than later. There is plenty of on-screen excitement to be had within those headsets, but the real drama of AR and VR may take place in the real world.
AR and VR: Some key players
Facebook
A couple of years back Mark Zuckerberg said he wanted to get one billion people to try VR. It's still a long way from happening. Facebook-owned Oculus, which it bought for $2 billion, offers three headsets: the all-in-one Oculus Go for VR viewing and Oculus Quest for VR gaming, and the Oculus Rift S, which needs to be connected to a PC, for high-performance gaming. Facebook just bought CTRL-Labs which could also help do away with the need for controllers in VR. "I don't think it's a 2020 thing. But hopefully it's not a 2030 thing," said Zuckerberg when asked when VR will hit the mainstream.
Apple
There are number of AR apps available for the iPhone, using its ARKit developer tools. These apps allow users to see virtual elements added to the view they can see through the smartphone screen. Ikea's Place app allows users to drop virtual furniture and other home furnishings into a room to see how they would look before buying them. Beyond this, Apple has long been rumoured to be working on some form of AR headset. "I'm incredibly excited by AR because I can see uses for it everywhere," Apple CEO Tim Cook said back in 2017.
Microsoft
Microsoft's HoloLens is a self-contained 'mixed reality' headset with wi-fi, and is aimed at business. Lift company ThyssenKrupp and Volvo have been among early users. Microsoft is preparing to launch Hololens 2, which sells for $3,500.
Google
Google has long been enthusiastic about AR and continues to pursue it in various directions. Google Glass is by far its best-known attempt at building a headset: while the consumer version met with ridicule, the Glass Enterprise Edition has been used by DHL, GE and Sutter Health to make staff more productive.
Within Android, Google has ARCore, a platform for building AR apps for its own Pixel smartphones, as well as handsets from Samsung, Huawei and others.
Google is also adding more AR features to its own apps. It recently added Live View to Google Maps so that when you can hold up your phone and see arrows and direction overlain on the view of the streets in front of you.
"Looking ahead, I believe that we are entering a new phase of computing: an era of the camera, if you will," said Aparna Chennapragada VP, Google Lens and AR last year.
Magic Leap
Magic Leap describes its headset as a 'spatial computer' that allows you to interact with digital content. "So when you make digital penguins walk off the edge of the coffee table, they fall off the edge of the coffee table, just like real penguins would," it says. The headset, which also requires you to carry a 'lightpack' that carries out the processing, sells for $2,295.
HTC
HTC has recently added to its Vive range of headsets with the HTC Vive Cosmos. This uses a set of six cameras to track the location of the headset and its user, rather than the laser-emitting base stations used previously. HTC recently worked with the BBC on an immersive Dr Who experience.
Read more on AR and VR
Mark Zuckerberg: VR isn't a 2020 thing, but hopefully isn't a 2030 thing
HTC appoints former Orange executive as new CEO
Qualcomm XR enterprise program blends vendor matchmaking and mixed reality
Mondly uses AR to help immerse you in language learning
Meltdown averted: How VR headsets are making nuclear power plants safer
The VR Linux desktop is on its way
Apple CEO Tim Cook touts enterprise augmented reality as Apple, SAP further integrate (TechRepublic)