The use of digital characters is growing fast not only in game development but also in animation for marketing and customer service. Nvidia's Audio2Face is one of the AI advances that has been speeding up the process of creating realistic facial animation synced to audio, and the GPU giant is now making it available to all.
Nvidia is open sourcing Audio2Face models and software development kit (SDK) so all game and 3D app developers can use it to build and deploy high-fidelity characters with cutting edge animations. That includes plugins for some of the best 3D modelling software and the best animation software.
Nvidia Audio2Face accelerates the creation of realistic digital characters by providing AI-driven real-time facial animation and lip-sync. The tool uses AI to generate realistic facial animations from audio input by analyzing acoustic features like phonemes and intonation to create a stream of animation data.
The data is then mapped to a character's facial poses. It can be rendered offline for pre-scripted content or streamed in real-time for dynamic, AI-driven characters, providing accurate lip-sync and emotional expressions.
The model is already used widely across gaming, entertainment and customer service. Game devs using it include Codemasters, GSC Games World, NetEase, Perfect World Games, while ISVs include Convai, Inworld AI, Reallusion, Streamlabs, The Farm 51 and UneeQ.
The software developer Reallusion has integrated it into its suite of tools, allowing integration with its iClone, Character Creator and iClone AI Assistant.
Now open source, the Audio2Face SDK includes libraries and documentation for authoring and runtime facial animations on-device or in the cloud. There are also plugins for Autodesk Maya and Unreal Engine 5, which allow users to send audio inputs and receive facial animation for characters.
Nvidia is also open sourcing the Audio2Face training framework, so anyone can fine-tune and customise pre-existing models for their specific use case.
Nvidia Audio2Face SDK can be downloaded from GitHub, where you'll also find instructions to build it. You can learn more on the Nvidia Developer site. You may need a computer with one of the best graphics cards to run it (see deals below).
