A yet-to-be-released app called Mixed Reality takes advantage of the ARKit augmented reality framework in iOS to insert VR gamers directly into their gameplay videos. Gamers recording their gameplay would otherwise need to use a green screen to splice themselves in after they record, using arduous video editing software. Most gamers who stream online use picture-in-picture to show the gameplay alongside a shot of them using their VR gear.
A good use of AR —
When it goes live (should Apple approve the use), Mixed Reality will work with the Oculus Quest or Quest 2, all-in-one VR headsets that can connect to an iOS device via Bluetooth. You'll also need an iPhone or iPad that features an A12 Bionic chip or newer. It looks like you will point your iOS device's rear camera at yourself as you play and the app will do all the work of putting you in the action.
Apple has spent years developing its AR framework to recognize the contours of objects in an environment so they can be captured as 3D objects and manipulated digitally. LIDAR in the new iPad Air improves on Apple's software work by adding infrared lasers to more accurately recognize the depth of objects in an environment. The developer behind Mixed Reality says it uses the "person segmentation" feature in ARKit that has been specifically trained to recognize people so they can be manipulated in different ways.
Virtual reality goggles like the Oculus Quest are the opposite of AR in that they place gamers into a captive digital environment. But they also take advantage of AR by using cameras to map the shape of a room so players are alerted when they're about to run into an object, and to track the position of the controllers for gameplay.
Apple's AR interests —
There have been rumors for years that Apple is planning on developing AR glasses, and its investments in the sensors and technology suggest it's possible if the company can get the hardware into a slim enough package.
Facebook is developing AR glasses and says it's been able to squeeze all the gadgetry into a pair of Ray-Ban glasses.
Apple CEO Tim Cook has said he thinks that AR could be the next major computing platform after the smartphone as it could enhance communication. Not many applications have yet been developed for AR besides fun face filters, however — and Apple's cute Memoji avatars. But if it really is the next big leap in computing it's no wonder the tech giants are racing to be first.
Mixed Reality is probably one of the coolest uses of AR thus far in iOS, and could make it really fun to record gameplay to upload to YouTube or stream to Twitch without needing to do much heavy lifting. If you're interested in signing up for the beta, there's a Google form here.
