Web AR
by Quince Gore-Rodney, Design Enterprise Studio Member, March 2021
Web-based Augmented Reality, aka WebAR, is one of the most exciting technologies at the forefront of web development. Augmented Reality (AR) is the connection of real life with a digital world, allowing us to extend reality through our computer devices. As a graduating student with an interest in 3D technologies such as three.js/react-three-fiber, I’m vested in the possibilities that augmented reality can do to 3D graphics and how it can give them more relevance to the real world.
While plenty of Augmented Reality use mobile app systems to bootstrap the technology like ARKit, ARCore, Vuforia etc. (Timchenko et al, p.211, 2017), they require downloading a separate application. WebAR content on the other hand, can be loaded instantly from the internet. This illusion of the physical world mixing combined with the ability of the internet, means WebAR provides instant immersion.
When WebAR was well in its infancy, it required web plugins such as ARToolkit, (like Adobe’s Flash Player we all had to download). These days, there are multiple ways to use WebAR. Recently Apple has announced support for AR content in the form of Quick Look, which is allowing users to download objects in the form of USDZ format to view in AR. But there is also another way.
Open-source, performance-light JavaScript libraries already exist that allow us to develop augmented-reality applications without use of plugins, no matter the internet device. AR.js, the brainchild of Jerome Etienne (2017) is based on a combination of three.js and A-Frame to use a camera to track images on the display, rendering 3D Graphics on top of the real-time video output.
Although AR is great for fun experiments and shopping previews, the technology also has as many practical uses as it has for entertainment. In times like these (global pandemic and all that) where it’s difficult to safely complete activities in busy spaces, WebAR can help bridge the gap for people working from home. Additionally, it could be used as an accessibility tool for the elderly or immuno-compromised.
Now, WebAR is not perfect. As an emerging technology it still has some setbacks, namely a memory limit on how much content it can load in a reasonable time (Koval, 2020) which is an issue for mobile devices. Hopefully, 5G should eliminate this setback.
All the future possibilities of WebAR are too long to list for this blog, but here are two examples which may help to explain why it offers unique new approach, one in the area of teaching, the other in entertainment:
Intangible Cultural Heritage through WebAR: Cantonese Porcelain
This case study on Chinese Cantonese Porcelain highlights how WebAR can be used to create embodied interactions, where the user relies on their senses to bridge their knowledge (like motion or touch gestures) with cultural heritage from the past (Tan, Ji, Hills, Feng., p.5, 2020). Unlike standard videos and images, emerging technologies like WebAR can be used better show and more importantly understand the “cultural contents of [Cantonese] heritage” (Tan, Ji, Hills, Feng., p.1, 2020).
For the Cantonese Porcelain WebAR study, a prototype was created using the ARToolkit plugin to track the camera position, and three.js and WebGL to bring assets to the 3rd dimension. Developing this application was a three-part setup; part one focusing on the classification of text, images, and video, and the second and third recording of materials and then developing a series of marker objects in a mobile interface for understanding the Cantonese Porcelain.
The researchers found that the AR prototype made activities more concrete and abundant, and by coupling historical facts with play they increased positive engagement from users. Personally, I think is great for the future of education, bridging the gap between education and entertainment. This could be re-purposed in at museum exhibitions or classes, as an accessible way to teach the public about more traditional subjects.
Unfortunately, this prototype isn’t publicly available, but the Case Study can be found here: https://dl-acm-org.ezproxy.uwe.ac.uk/doi/pdf/10.1145/3334480.3375199.
Soccer AR/ Soccer On the Tabletop
Augmented Reality doesn’t have to be used as a tool on its own and one of the best demonstrations of this on the internet is Soccer AR (McKaskill, 2018). This system combines two cutting edge technologies, Artificial Intelligence (Machine Learning) and Cloud-Computing to display a real-time match of football (or Soccer as it is called across the Atlantic) in Augmented Reality. Its dream come true for football fans and techies alike.
This experiment, a collaboration between the University of Washington, Google and Facebook, estimates the depth map of each player, based on 3D player data from video games (Rematas, Kemelmacher-Shlizerman, Curless, Seitz., 2018) and compares it by estimating pose and depth using AI (Machine Learning) techniques with YouTube soccer footage to create an animated scene.
Unlike other AR experiments, Soccer On the Tabletop uses a head-mounted display called HoloLens which combines Augmented Reality with Virtual Reality (a technology that completely immerses the user in a digital world) to create a mixed-reality where the real and virtual world are intertwined. This is a rare application of a future technology called WebXR which has only just come into realisation in recent years. Pretty much next-level stuff!
If you would like to see the AR experiment working or download the source code, here is the official page: https://grail.cs.washington.edu/projects/soccer/ or check out the YouTube link to see it in action: https://www.youtube.com/watch?v=eRGAB4QBS6U.
In summary, I believe that WebAR is an exciting technology which brings new possibilities for creating innovative experiences and gives a level of relevance to 3D web technologies that we have never been seen before. The value of using the context of an environment to render display interactive objects, combined with the instantaneous loading the internet brings, gives it countless possibilities of how it could be used in different scenarios and cases.
Despite the current setbacks like internet speeds and file size which limits its potential, WebAR will soon become more accessible as mobile internet and optimisations improve. In the next coming years, when more AR content comes to the web, experiences such as Soccer AR will become a part of our everyday lives, integrating itself into the myriad of technologies we already use every day.
Bibliography:
A-Frame (2020) A-Frame-A web framework for building 3D/AR/VR experiences. Available from: https://aframe.io/ [Accessed 12th March 2021]
AR.js (2020) AR.js Documentation. Available from: https://ar-js-org.github.io/AR.js-Docs/ [Accessed 12th March 2021]
Etienne, J. (2017) AR.js. Available from: https://github.com/AR-js-org/AR.js [Accessed 29th Febuary 2021]
Koval, R (2020) WebAR: How It’s Changing the Future of Augmented Reality. Perfectial. [blog] Available from: https://perfectial.com/blog/webar/ [Accessed 12th March 2021]
McCaskill, S. (2018) Soccer On Your Tabletop [image] In: Mckaskill, S. (2018) Soccer On Your Tabletop: Will We Be Watching The Next World Cup in AR? Forbes. Available from: https://www.forbes.com/sites/stevemccaskill/2018/06/27/soccer-on-your-tabletop-will-we-be-watching-the-next-world-cup-in-ar/?sh=6b83d253e380 [Accessed 12th March 2021]
Rematas, K., Kemelmacher-Shlizerman, Ira., Curless, B., Seitz, S. (2018) Soccer on Your Tabletop. In: CVPR 2018. Editor Unknown. 2018 Conference on Computer Vision and Pattern Recognition. Salt Lake City. June 18-22, 2018. Washington, University of Washington. Available From: https://grail.cs.washington.edu/projects/soccer/ [Accessed 12th March 2021]
Three.js (2020) three.js. Available from: https://threejs.org/ [Accessed 26th February 2021]
Timchenko, R., Grechnyev, O., Skuratovskvi, S., Chyrka, Y., Gorovyi, I. (2020) Augmented Reality in Web: Results and Challenges ed. IEEE Third International Conference on Data Stream Mining & Processing. August 21-25, 2020, Lviv, Ukraine. New York City, IEEE. Available from: https://ieeexplore-ieee-org.ezproxy.uwe.ac.uk/document/9204240 [Accessed 12th March 2021]
XR Today: https://www.xrtoday.com/mixed-reality/why-is-hololens-good-for-mixed-reality/