The Anatomy of a Plug
The initial Plug and Plug Frame format is not efficient.
It was put together over time, and in a scramble, to understand which properties were meaningful and to assist with testing Plug playback. Additionally, since we initially prioritized realtime human-in-the-loop, the decision was made to allow the MindTwin to receive and play back individual frames, rather than a full sequence.
The Plug is meant to act as a container for a set of Plug Frames. The Plug should contain useful metadata that can be used to play back the contained frames. Even audio. Unfortunately, in order to facilitate a realtime stream of frames, the concept of using the Plug as a header format was suppressed in favor of sending metadata on every frame. This made the Plug header less useful and Plug Frames more aware and stable, but at a huge data cost.
In the MindTwin project, Plug Frames are denoted with this icon
In the MindTwin project, full Plugs are denoted with this icon
Steps to improve the efficiency of the format
Require an initial Plug
This will eliminate the need to send metadata on every frame
This will enable the newer style of optimized property curves in the future (see below)
Omit properties that aren’t being used
Currently there are about forty properties on the Plug Frame packet, but only a small handful are used
This can be pruned quickly
On the receiving end, a Plug Frame is always initialized with a set of defaults. The incoming packet alters those default values to the values on the Plug Frame. If a property isn’t used, it will automatically use default values.
Constant key reduction / Only send changes in values
If the value doesn’t change on the next frame, don’t send it and don’t change it
One of the benefits of the current system is that you don’t need to send data in order to compose a frame with default settings, but we could simply explain in the Plug that a particular sequence is expected to consume last frame’s property values, if available.
Don’t send named properties
Initially, named properties were used to be compatible with the internal Unity PlugFrame object. But thanks for Brian’s refactor, this system has been split into a PlugFrame and PlugFramePacket class. The PlugFramePacket class acts as a json intermediary to the PlugFrame and does not need to be human readable or accessed directly. The PlugFrame class contains helper methods for interacting with PlugFramePacket data. As a result, it would be more efficient to send something like coma separated values or an array that could include null values.
Forced Multi-Property Selection
In some cases, there are multiple ways to interact with a Plug Frame. For eyebrows, for instance, there are a total of 8 properties designed to be used in one of two ways. Two of those properties are for general raising and lowering and are currently the only eyebrow properties used. But the other 6 properties enable finer control over eyebrow articulation, three controls for each eyebrow. As an example, by flagging and restricting usage of these types of convenience properties, we could send only 2 or 6 sets of data per frame, rather than 8 - always.
Property Curves - The Future of Plugs
As part of a redesigned Plug format, I’ve been experimenting with something I’m calling Property Curves. The efficiency of Plug Frame data can be increased further by using tangent curves rather than per-frame properties.
Since we can sample a curve at any point throughout a sequence, we not only are able to store data in a more efficient way, but have much smoother playback of ‘in-between’ frames. Because the Plug Controller is designed to flexibly play back Plug Frames smoothly, if the playhead finds itself at a sampling point that doesn’t have a directly correlated frame, it will mix the properties between the previous and next frame to avoid stepping.
Consider a situation where a MindTwin rotates their head from fully left to fully right over the course of 100 frames. If that was the only property we wanted to send, the json for that single property would look something like this:
{0: {'head_rotation_no': 0.0}, 1: {'head_rotation_no': 0.010101010101010072}, 2: {'head_rotation_no': 0.020202020202020225}, 3: {'head_rotation_no': 0.030303030303030297}, 4: {'head_rotation_no': 0.04040404040404045}, 5: {'head_rotation_no': 0.05050505050505052}, 6: {'head_rotation_no': 0.060606060606060594}, 7: {'head_rotation_no': 0.07070707070707066}, 8: {'head_rotation_no': 0.08080808080808081}, 9: {'head_rotation_no': 0.09090909090909088}, 10: {'head_rotation_no': 0.10101010101010104}, 11: {'head_rotation_no': 0.1111111111111111}, 12: {'head_rotation_no': 0.12121212121212119}, 13: {'head_rotation_no': 0.13131313131313133}, 14: {'head_rotation_no': 0.1414141414141414}, 15: {'head_rotation_no': 0.15151515151515152}, 16: {'head_rotation_no': 0.16161616161616163}, 17: {'head_rotation_no': 0.1717171717171717}, 18: {'head_rotation_no': 0.18181818181818182}, 19: {'head_rotation_no': 0.19191919191919188}, 20: {'head_rotation_no': 0.202020202020202}, 21: {'head_rotation_no': 0.2121212121212121}, 22: {'head_rotation_no': 0.2222222222222222}, 23: {'head_rotation_no': 0.23232323232323232}, 24: {'head_rotation_no': 0.24242424242424246}, 25: {'head_rotation_no': 0.25252525252525254}, 26: {'head_rotation_no': 0.26262626262626265}, 27: {'head_rotation_no': 0.2727272727272727}, 28: {'head_rotation_no': 0.2828282828282828}, 29: {'head_rotation_no': 0.29292929292929293}, 30: {'head_rotation_no': 0.30303030303030304}, 31: {'head_rotation_no': 0.31313131313131315}, 32: {'head_rotation_no': 0.32323232323232326}, 33: {'head_rotation_no': 0.3333333333333333}, 34: {'head_rotation_no': 0.3434343434343434}, 35: {'head_rotation_no': 0.35353535353535354}, 36: {'head_rotation_no': 0.36363636363636365}, 37: {'head_rotation_no': 0.37373737373737376}, 38: {'head_rotation_no': 0.38383838383838376}, 39: {'head_rotation_no': 0.3939393939393939}, 40: {'head_rotation_no': 0.404040404040404}, 41: {'head_rotation_no': 0.41414141414141414}, 42: {'head_rotation_no': 0.4242424242424242}, 43: {'head_rotation_no': 0.43434343434343436}, 44: {'head_rotation_no': 0.4444444444444444}, 45: {'head_rotation_no': 0.45454545454545453}, 46: {'head_rotation_no': 0.46464646464646464}, 47: {'head_rotation_no': 0.47474747474747475}, 48: {'head_rotation_no': 0.4848484848484849}, 49: {'head_rotation_no': 0.494949494949495}, 50: {'head_rotation_no': 0.5050505050505051}, 51: {'head_rotation_no': 0.5151515151515151}, 52: {'head_rotation_no': 0.5252525252525253}, 53: {'head_rotation_no': 0.5353535353535354}, 54: {'head_rotation_no': 0.5454545454545454}, 55: {'head_rotation_no': 0.5555555555555556}, 56: {'head_rotation_no': 0.5656565656565656}, 57: {'head_rotation_no': 0.5757575757575758}, 58: {'head_rotation_no': 0.5858585858585859}, 59: {'head_rotation_no': 0.5959595959595959}, 60: {'head_rotation_no': 0.6060606060606061}, 61: {'head_rotation_no': 0.6161616161616161}, 62: {'head_rotation_no': 0.6262626262626263}, 63: {'head_rotation_no': 0.6363636363636364}, 64: {'head_rotation_no': 0.6464646464646465}, 65: {'head_rotation_no': 0.6565656565656566}, 66: {'head_rotation_no': 0.6666666666666666}, 67: {'head_rotation_no': 0.6767676767676768}, 68: {'head_rotation_no': 0.6868686868686869}, 69: {'head_rotation_no': 0.696969696969697}, 70: {'head_rotation_no': 0.7070707070707071}, 71: {'head_rotation_no': 0.7171717171717172}, 72: {'head_rotation_no': 0.7272727272727273}, 73: {'head_rotation_no': 0.7373737373737373}, 74: {'head_rotation_no': 0.7474747474747475}, 75: {'head_rotation_no': 0.7575757575757577}, 76: {'head_rotation_no': 0.7676767676767675}, 77: {'head_rotation_no': 0.7777777777777778}, 78: {'head_rotation_no': 0.7878787878787878}, 79: {'head_rotation_no': 0.7979797979797981}, 80: {'head_rotation_no': 0.808080808080808}, 81: {'head_rotation_no': 0.8181818181818182}, 82: {'head_rotation_no': 0.8282828282828283}, 83: {'head_rotation_no': 0.8383838383838383}, 84: {'head_rotation_no': 0.8484848484848484}, 85: {'head_rotation_no': 0.8585858585858586}, 86: {'head_rotation_no': 0.8686868686868687}, 87: {'head_rotation_no': 0.8787878787878788}, 88: {'head_rotation_no': 0.8888888888888888}, 89: {'head_rotation_no': 0.898989898989899}, 90: {'head_rotation_no': 0.9090909090909091}, 91: {'head_rotation_no': 0.9191919191919192}, 92: {'head_rotation_no': 0.9292929292929293}, 93: {'head_rotation_no': 0.9393939393939394}, 94: {'head_rotation_no': 0.9494949494949495}, 95: {'head_rotation_no': 0.9595959595959596}, 96: {'head_rotation_no': 0.9696969696969698}, 97: {'head_rotation_no': 0.9797979797979799}, 98: {'head_rotation_no': 0.98989898989899}, 99: {'head_rotation_no': 1.0}}
Yikes. That’s a ton of data - particularly because we send named properties down on each frame. Uncompressed, it’s 4589 bytes.
The same set of data can be expressed with only 2 keyframes. In this case, with or without additional tangent data to define curvature. In this example, each array element contains a frame number and a normalized Plug Frame value.
{'head_rotation_no': [[0, 0.0], [100, 1.0]]}
A measly 44 bytes! A savings of over 99%.
While this system is still experimental, it’s written and currently functioning. In the current MindTwin project, pressing keys 1-5 will play a Local Plug, composed with Property Curves. The other half of this implementation would require testing the efficiency of data compression and writing a compiler that can intelligently decide the most efficient way to pack a Property Curve.
Property Curve Risks
Noisier data is less efficient than per-frame properties
Unless keyframes are stored as two values without tangents, which is acceptable for noisy data
This system complicates chunked streaming since curves need to remain intact and live on the Plug object, not Plug Frame
It’s possible that the reduction of data is worthwhile and can eliminate the need to chunk
It’s possible that we shift to a system that expects fixed time packets to allow for chunking. Ex. Every Plug packet is 1 second in duration…