Historically there has been a clear distinction between Augmented Reality (AR) headsets and Virtual Reality (VR) headsets however with recent developments in VR technology the two are starting to overlap in Mixed Reality (MR) applications. This case study will look at the what these advancements are and how it effects the user experience in MR environments.
Introduction
Traditionally VR headsets have focused on an experience that is disconnected from the real and when wearing the headset the outside world is isolated from the virtual by blocking outside light, a rendered scene replaces the real, and a surround sound environment is added engulfing the senses in a new world. More recently though, cameras mounted on the headset have allowed the real world to be brought into the virtual (fig. 1) and as the technology has evolved especially in the realm of stand alone (untethered) headsets are now offering a Mixed reality experience that is comparable to that of the much more expensive AR based headsets like the Hololens or Magic Leap. This technique of using cameras to bring the real and virtual together is commonly referred to as ‘Passthrough’ mixed reality.
Headset Design
To get where we are today a lot of ‘hacking’ has had to take place to create a mixed reality experience within a headset that was originally design to replace reality, and in the process has produced a product that is not only comparable to the AR based headsets but in many ways offers features that surpass it. It may seem relatively simple, just fit some cameras to the front of the headset and pass the image to the displays in the headset, however the problem is the passthrough cameras cannot be physically co-located with the user’s eyes (Xiao et al, 2022). A camera too far from the eye may cause an inability to interact correctly with the real environment around the user and may display real objects in the wrong location (Larroque, 2021). In addition to this, a camera lens wide enough to capture the image will have lens distortions that overlap with its neighbouring camera and will need to be corrected. To overcome this, complex algorithms have been developed (fig. 2) to recreate the view the wearer would see if the headset was absent however this creates a significant issue, calculating such a complex algorithm (in addition to rendering additional content) takes processing time, which = latency, which is bad. If there is a delay between adjusting your gaze and seeing the resulting image the wearer will likely feel an order of discomfort through to extreme nausea. The code also has to be as efficient as possible as to not hinder the hardware’s ability to process other data (esp. the rendered content), or consume to much energy as to over heat the headset and/or impact battery life. This has been one area of development that has required a significant amount of research and could be considered the holy grail of passthrough technology. Meta Reality Labs who purchased Oculus in 2014 have approximately 17,000+ Employees (Schiffer, 2022) working on VR based headsets and reported a loss in 2022 alone of $13.72 billion USD (Vanian, 2023).
The Meta Quest Pro.
Recently released headsets offer a much more refined than there predecessors, and the Meta Quest Pro, which features colour stereo passthrough, and a design that allows for both VR and MR applications all in a standalone headset that can offer an experience with true colour virtual elements that [almost] seamlessly meld with the real in a package that is comfortable and feature packed. Their significant investment has resulted in a headset that is addressing a lot of the disadvantages of VR based headsets when compared to AR based headsets, for example, improved peripheral vision, an accurate rendering of the real world, and excellent processing power for a standalone device. These advancements have lead to a vastly improved user experience however there has been a significant increase in the cost of the headset @ $1,960 NZD when compared to their cheaper Quest 2 (from $710) which is capable of a rather crude black & white MR environment To note though this is still much cheaper than the Hololens 2 @ $6,200 NZD and comparable in price to the tethered Vive Pro 2 @ $1,600 NZD which requires a high end PC to drive it and has limited MR applications.
Software
As passthrough headsets are based on VR technology there is a much larger user base due to VR having a strong foothold in the consumer market. This results in a platform with a large number of end users and developers which intern promotes innovation while reducing total cost of ownership. AR headsets have a much higher cost of ownership and thus has resulted in an industry that is focused on enterprise clients which is a much smaller market and hence more suited to those wishing to develop solutions ‘in house’ for specific tasks. This difference in target market may have a bigger influence on the adoption and development of this technology over anything else simply because it more accessible.
Conclusion and recommendations
As the design of passthrough MR headsets develop the shortcomings diminish. Processing power, efficiency, optics, display technology, cameras, software and engineering are all contributing to a much improved and capable product. The fact that the headset is physically blocking the real world does bring up safety concerns especially in dangerous environments such as construction, industrial installations, and aviation and they may never replace AR based solutions in these situations however they do offer a lot of significant advantages especially as the can cover the full spectrum of XR experiences, are much more affordable have a larger user base and a significant software library. In saying that if you’re developing a solution you’ll need to weigh up whether the real environment or the rendered content is the focus of your project as that will ultimately be the deciding factor.
References
Xiao, L., Nouri, S., Hegland, J., Garcia Garcia, A., & Lanman, D. (2022). Neural Passthrough: Learned Real-Time View Synthesis for VR. ACM SIGGRAPH 2022 Conference Proceedings. https://doi.org/10.1145/3528233.3530701
Larroque, S. (2021). Digital Pass-Through Head-Mounted Displays for Mixed Reality. Information Display, 37(4), 17-21. https://doi.org/10.1002/msid.1228
Chaurasia, G., Nieuwoudt, A., Ichim, A.-E., Szeliski, R., & Sorkine-Hornung, A. (2020). Passthrough+: Real-time Stereoscopic View Synthesis for Mobile Mixed Reality. Proceedings of the ACM on Computer Graphics and Interactive Techniques, 3(1), 1-17. https://doi.org/10.1145/3384540
Schiffer, Z. (2022, May 18). Meta imposes hiring freeze on commerce, Messenger Kids, and Facebook Gaming. The Verge. https://www.theverge.com/2022/5/18/23125571/meta-hiring-freeze-commerce-messenger-kids-facebook-gaming
Vanian, J., & Levy, A. (2023, February 1). Meta lost $13.7 billion on Reality Labs in 2022 after metaverse pivot. CNBC. https://www.cnbc.com/2023/02/01/meta-lost-13point7-billion-on-reality-labs-in-2022-after-metaverse-pivot.html