Pupil movements, body poses and nose scrunching are among the flickers of human expression that Meta wants to harvest in building its metaverse, according to an analysis of dozens of patents recently granted to Facebook’s parent company.
Chief executive Mark Zuckerberg has pledged to spend $10bn a year over the next decade into the nebulous and much-hyped concept denoting an immersive virtual world filled with avatars. Rivals such as Apple and Microsoft are also pursuing similar aims that Big Tech executives describe as part of the next evolution of the internet.
The Financial Times has reviewed hundreds of applications to the US Patent and Trademark Office, many of which were granted this month. They reveal that Meta has patented multiple technologies that wield users’ biometric data in order to help power what the user sees and ensure their digital avatars are animated realistically.
But the patents also indicate how the Silicon Valley group intends to cash in on its virtual world, with hyper-targeted advertising and sponsored content that mirrors its existing $85bn-a-year ad-based business model.
This includes proposals for a “virtual store” where users can buy digital goods, or items that correspond with real-world goods that have been sponsored by brands.
“For us, the business model in the metaverse is commerce-led,” Nick Clegg, Meta’s head of global affairs told the FT during a recent interview. “Clearly ads play a part in that.”
The patents do not mean that Meta will definitely build the technology, but they offer the clearest indication yet of how the company aims to make its immersive world into a reality.
Some of the patents relate to eye and face tracking technology, typically collected in a headset via tiny cameras or sensors, which may be used to enhance a user’s virtual or augmented reality experience. For example, a person will be shown brighter graphics where their gaze falls, or ensuring their avatar mirrors what they are doing in real life.
One Meta patent, granted on January 4, lays out a system for tracking a user’s facial expressions through a headset that will then “adapt media content” based on those responses.
There is a “wearable magnetic sensor system” to be placed around a torso for “body pose tracking”. The patent includes sketches of a user wearing the device but appearing in virtual reality as a soldier complete with a sword and armour.
Another patent proposes an “avatar personalisation engine” that can create three dimensional avatars based on a user’s photos, using tools including a so-called skin replicator.
“Meta aims to be able to simulate you down to every skin pore, every strand of hair, every micromovement,” said Noelle Martin, a legal reformer who has spent more than a year researching Meta’s human-monitoring ambitions with the University of Western Australia.
“The objective is to create 3D replicas of people, places and things, so hyper-realistic and tactile that they’re indistinguishable from what’s real, and then to intermediate any range of services . . . in truth, they’re undertaking a global human-cloning programme.”
The project has allowed the company, which in recent times has been stung by other scandals over moderation and privacy, to attract engineers from rivals such as Microsoft amid a fierce battle for talent between the world’s biggest technology companies.
Since changing its name from Facebook to Meta in late October in a corporate rebranding, the company’s share price has risen about 5 per cent to $329.21.
Critics remain sceptical of the vision, suggesting the effort is a distraction from recent scrutiny after whistleblower Frances Haugen last year publicly accused the company of prioritising profit over safety*.
“What are they going to do with more data and how are they going to make sure it is secure?” said Celia Hodent, former director of user experience at Epic Games who now works as an independent consultant.
Some patents appear focused on helping Meta with its ambitions to find new revenue sources amid concern over fading interest with younger users in its core social networking products such as Facebook.
Zuckerberg has indicated the company plans to keep the prices of its headsets low, but instead draw revenues in its metaverse from advertising, and by supporting sales of digital goods and services in its virtual world.
One patent explores how to present users with personalised advertising in augmented reality, based on age, gender, interest and “how the users interact with a social media platform”, including their likes and comments.
Another seeks to allow third parties to “sponsor the appearance of an object” in a virtual store that mirrors the layout of a retail store, through a bidding process similar to the company’s existing advertising auction process.
The patents indicate how Meta could offer ads in its immersive world that are even more personalised than what is possible within its existing web-based products.
Research shows that eye gaze direction and pupil activity may implicitly contain information about a user’s interests and emotional state, for example, if a user’s eyes linger over an image, this may indicate they like it.
“Clearly, you could do something similar [to existing ad targeting systems] in the metaverse — where you’re not selling eye-tracking data to advertisers, but in order to understand whether people engage with an advertisement or not, you need to be able to use data to know,” Clegg said.
Brittan Heller, a technology lawyer at Foley Hoag, said: “My nightmare scenario is that targeted advertising based on our involuntary biological reactions to stimuli is going to start showing up in the metaverse . . . most people don’t realise how valuable that could be. Right now there are no legal constraints on that.”
Meta said: “While we don’t comment on specific coverage of our patents or our reasons for filing them, it’s important to note that our patents don’t necessarily cover the technology used in our products and services.”
Additional reporting by Henry Mance in London
Copyright The Financial Times Limited 2022
