While the cinematic may not appear too different to the original, in its final rendered version the integration of Ziva technology has brought a new dimension to our protagonist.

Ziva brings decades of experience and pioneering research from the VFX industry to enable greater animation quality for games, linear content production, and real-time projects. Its machine learning-based technology helps achieve extraordinary realism in facial animation, and also for body and muscle deformations.

To achieve the level of realism in Enemies, Ziva used machine learning and 4D data capture – going beyond the traditional process of scanning actors in 3D scans.

The static, uneditable 4D captured facial performance has now been transformed into a real-time puppet with a facial rig that can be animated and adjusted easily at any time – all while maintaining the high fidelity of the original 4D captured performance.

Our team built on that 4D capture data and trained a machine-learned model that could be animated to create any performance. The end result is a 50MB facial rig that has all the detail of the 4D captured performance, without the need to carry the weight of the original 3.7GB of the 4D capture data.

This technology means that you can replicate the results with a fraction of the animation data, and you can affect the results in real-time in a way that 4D doesn’t allow.

In order to achieve this, Unity’s Demo Team did the following:

Creating the puppet

  • To create this new version of Louise, we worked with the Ziva team who handled the machine learning workflow using a pre-existing 4D data library. Additional 4D data was collected from a new performance by the original Enemies actor; we only needed to collect a few additional expressions. This is one of the unique advantages to our machine learning approach.
  • With this combined dataset, we trained a Ziva puppet to accurately reproduce the original performance. We could alter this performance in any way, ranging from tweaking minute details to changing the entire expression.
  • Using the 4D capture data through machine-learning software, this can enable any future performance to run on any 3D head by showing a single performance applied to multiple faces of varying proportions. This makes it easier to expand the range of performances to multiple actors and real-time Digital Humans in any future editions.

The puppet’s control scheme

  • Once the machine learning was completed, we had 200-300 parameters that –  when used in combination and at different weights – could recreate everything we had seen in the 4D data, with incredible accuracy. We didn’t have to worry about a hand-animated performance looking different when used by a group of different animators. The persona and idiosyncrasies of the original actor could come through no matter how we chose to animate the face.
  • As Ziva is based on deformations and not an underlying facial rig, this allowed us to manipulate even the smallest detail because the trained face uses a control scheme that was developed to take advantage of the fidelity of the machine learned parameters/data.
  • At this point, creating a rig is a pretty flexible process as we can just tap into those machine learned parameters; this in turn deforms the face. There are no joints in a Ziva puppet, aside from basic logical face and neck joints. 

So what does this all mean?

There are many advantages to this new workflow. First and foremost, we now have the ability to dynamically interact with the performance of the digital human in Enemies.

This allows us to change the character’s performance after it has been delivered. Digital Louise can now say the same lines as before, but with very different expressions. For example, she can be friendlier or angrier or convey any other emotion that the director requires.

We are also able to manually author new performances with the puppet – facial expressions and reactions that the original actress never performed. If we wanted to develop the story into an interactive experience, it would be important to be able to expand the possibility for the digital character to react to, for instance, a player’s chess moves with approval or disapproval.

For the highest level of fidelity, the Ziva team can create a new puppet with its own 4D data set. Ziva also recently released a beta version of Face Trainer, a product built on a comprehensive library of 4D data and ML algorithms that trains any face mesh to perform the most complex expressions in rea-ltime with no need for new 4D capture.

Further, it is possible to create entirely new lines of dialogue, all at a fraction of the time and cost that the creation of the first line required. We can do this either by getting the original actress to perform additional lines with an HMC and then use the HMC data to drive the puppet, or by getting another performer to deliver the new lines and retargeting their HMC data to the existing puppet.

Source: Unity Technologies Blog