PlugFrame's have an eye openness property for left/right eyes. This openness value sets the eyelid openness from 0.0-1.0, where 0.5 is resting, 0.0 is closed and 1.0 is open wide. With great enough tracking precision, eye openness can also convey blinking on the same property since an eyelid can only be in a fixed number of configurations.

Shader Mishap = American Horror Story Promo

If we wanted the most detailed, most accurate per-frame reconstruction of a subject’s mouth, how might we go about capturing that?

My dream scenario for capturing and translating accurate human speech looks something like a (nearly) per-frame detailed point cloud. From there, we could annotate and solve tracking issues in 3D, as well as have access to the detailed curvature of the lips and mouth region - all helping to inform our rig on how to drive the render mesh. In some cases, these points could directly drive the render mesh.

Assuming that this type of detailed information is useful, how might one go about capturing such a massive amount of data?