There's been a long-standing issue with serialization in the MindTwin project. Since the project uses mostly custom classes for various parts of the rig, each component's serialization was creating needless slowdowns during playback - particularly in Plug and PlugFrame classes and their dependencies.

I’ve spent some time updating Lars’ MTP - an often neglected puppet! More updates to follow, but he’s now been upgraded to use the latest systems along with the other puppets. He has new eyes & eyelids, as well as upgraded mouth controls. His project fell a little out of date since he was one of the first experimental puppets to be made. More updates to follow.

There’s a strange bug that I need to track down that occurs when a Plug mixes in and out. You can see the strange mixing in this video.

PlugFrame's have an eye openness property for left/right eyes. This openness value sets the eyelid openness from 0.0-1.0, where 0.5 is resting, 0.0 is closed and 1.0 is open wide. With great enough tracking precision, eye openness can also convey blinking on the same property since an eyelid can only be in a fixed number of configurations.

Shader Mishap = American Horror Story Promo

If we wanted the most detailed, most accurate per-frame reconstruction of a subject’s mouth, how might we go about capturing that?

My dream scenario for capturing and translating accurate human speech looks something like a (nearly) per-frame detailed point cloud. From there, we could annotate and solve tracking issues in 3D, as well as have access to the detailed curvature of the lips and mouth region - all helping to inform our rig on how to drive the render mesh. In some cases, these points could directly drive the render mesh.

Assuming that this type of detailed information is useful, how might one go about capturing such a massive amount of data?

Here’s a quick clip of Obama, Mandela and Serkis playing the same Plug. This is the first version of a new Plug system that uses curves to build the individual sequence PlugFrames. It also uses the Speech Engine 3.0 system, which is still a work in progress.

Until recently, the eye blink system was fairly basic. It involved two animation cycles, one for eyelids closing and one for eyelids opening. The first cycle was timed to be slightly shorter than the first and they both used basic ease in/out curves.

But ultimately the results were as basic as the above description.

Eye blinking is actually reasonably complex and fairly variable in nature. I came across some research that helped influence the newer eye blink system.

This system is designed to use the curvature of the lips to affect the skin around the face, as is common in human speech. Although it’s a series of muscles that do the deformations in an actual human, lip curvature is used in our system to simulate this movement.

During MTP setup, each vertex on the head mesh is assigned a sibling outer lip point. This is essentially each point’s nearest outer lip point….