Are you curious about how Oats Studios created such high-fidelity sets, characters and costumes in real time for ADAM: The Mirror and ADAM: Episode 3? Read our in-depth, behind-the-scenes blog posts on lighting, Alembic support, clothing simulation, shaders, real-time rendering and more.
I'm SerEn (Sean) Low, an Art Tools Engineer specialized in Tools and Pipeline on the Made with Unity team. Prior to joining Unity Technologies, I was at Disney Interactive, developing animation tools for their console games, which collaborated with Pixar, Marvel and Lucasfilm. For the ADAM films, I helped integrate tools such as Alembic and Timeline, making sure the animation pipeline is streamlined and easy to maintain.
Background: The birth of Alembic in Unity
About two years ago, the Unity R&D team initiated the Alembic Importer plugin. It was used in a short film named The Gift by Marza Animation Planet. In one of the most challenging shots in the film, there are millions of plastic balls that form an ocean wave, which is done by importing the Alembic data generated from a Houdini simulation into Unity. It was a solid proof point that Alembic could be integrated into Unity by allowing artists to bring their simulated data from various external applications. Alembic also allows artists to break through the limitations of joint-based mesh deformation in a real-time engine.
When Oats Studios decided to use our Alembic Importer in their pipeline, they also wanted additional functionality and optimization so that it could handle high visual fidelity for cloth simulation and real-life facial-performance capture. Since the requirement was vastly different from what the Alembic Importer was originally developed for, Unity's Dev team put in a huge effort to upgrade this plugin.
Major upgrades for the Alembic Importer
The first major update of the Alembic Importer was Timeline Integration. This allows Alembic data to stream in and playback through Timeline during runtime. Simultaneously, during edit time, the user can scrub through Timeline while viewing the Alembic playback. To enable this integration, we added a custom PlayableBehavior (Alembic Stream Player), PlayableAsset (Alembic Shot Asset) and TrackAsset (Alembic Track) to the Alembic Importer package. (For details about the Playables API, check out the documentation.)
In Timeline, you can then add an Alembic Track, create an Alembic Shot Asset and associate it with the Alembic instance in-scene with an Alembic Stream Player component attached.
An Alembic Track and Alembic Shot Asset in Timeline
An Alembic Stream Player component in scene object
The second significant update for the Importer was support for Vertex Sharing, which was important to optimize Alembic streaming performance for constant mesh topology assets. While typical Alembic data is used for dynamic simulation such as particles (where mesh topology is always changing), we wanted to merge shared vertices in order to optimize the playback performance. This is highly important since the mesh is reconstructed every frame based on the data streaming in real time. By merging the shared verts, other mesh properties such as edge, UV, normal and tangent are updated too, so the dynamic Alembic mesh now behaves like an FBX mesh. This ensures the mesh reacts according to assigned materials, shaders and lighting in a scene.
Alembic mesh in Unity showing shared vertices
For the third major update, we added Interpolation in Alembic vertex data. Since Alembic mesh is basically a new mesh constructed every frame, it doesn't have the information about the delta of the previous and next vertex position. This is an issue when you try to apply motion blur onto Alembic mesh for rendering. With interpolation enabled, the Alembic asset in Unity is able to calculate the interpolation between positions and thus output the velocity that is necessary to calculate motion vectors. That means you can now have accurate motion vectors derived from the Alembic mesh to be used in shader computation or by the Post-Processing Stack. This enhances motion blur quality in final rendered images and it also improves performance because fewer samples are needed.
Motion vector on Alembic mesh
Before motion blur applied
After motion blur applied
The motion vector buffer
Additionally, several other production-friendly features were implemented to enable a smooth production workflow, such as auto-activate on Alembic Playable Asset, which enables the activation of the Alembic Playable Asset without the need of an additional Activation Track in Timeline. The Time Scale property allows you to re-time the playback rate of Alembic data. The Turn Quad Edge property allows you to choose which orientation the mesh faces should triangulate in. There is also an option to cache all Alembic samples whenever the Alembic scene object is enabled, instead of having to cache the data during playback.
Alembic in production at Oats Studios
"Alembic streaming was a very significant aspect of the project for us. Bringing simulated things over is difficult and the Alembic Importer in Unity is a game-changer," said Oats Studios' VFX Supervisor, Chris Harvey.
Assembling multiple Alembic assets in Timeline editor
Based on real wardrobes, all the clothing in Oats' ADAM films was captured by photogrammetry and then recreated and simulated in Marvelous Designer, where digital tailor Sean Frandsen applied real physical attributes and animation to the different fabrics (wool, cotton, leather, etc.).
Meanwhile, facial performance was captured and converted into a series of static meshes for Alembic playback. (For more information about facial capture, watch the ADAM: Behind the Scenes video.)
In ADAM, Alembic is mainly used for two types of assets: Cloth and faces
The data was then imported into Autodesk Maya for further cleanup and tweaking before being exported out as an Alembic file. Here's a schematic of their pipeline.
The Alembic and FBX assets pipeline into Unity
Assembling simulated assets in space and time
At this stage in Unity, Oats used Timeline as an assembly tool, combining Alembic assets and choreographing them in scene space along with other assets. For example, in order to fully assemble a full character performance, they used two Alembic tracks, one each for facial and cloth, on top of an Animation track, which contains body performance in FBX format as shown below.
The assembled character comes to life in Timeline
With everything in Timeline, Oats could synchronize and sequence all asset performances according to each shot's timing for real-time playback. However, large numbers of assets can easily overcomplicate the Timeline editor interface and be hard to maintain once in production. Oats solved this by organizing assets in-scene using TrackGroup, which also let them expand or collapse groups as needed. Another Unity feature that Oats tapped to keep things well organized and manageable was ControlTrack for handling multiple nested Timeline instances.
Expanded and collapsed Track Groups
ControlTracks handle nested timeline assets (Characters and Caches shown here)
Later in the production stage, the amount of generated Alembic data for both facial mocap and cloth simulation reached almost 300 files. Therefore Oats needed an automated process to import multiple Alembic files into Timeline, so they implemented their own importer to automatically bring in Alembic files with predefined settings, instantiate them into a scene with the required components as well as sequence them in Timeline. With help from the Unity Dev team, Oats was able to kick-off this automation process with the click of a button. We also gathered a lot of great feedback and API requests from this project to further improve Timeline.
Managing hundreds of Alembic assets for various characters and props
Over the course of the ADAM project, the Alembic Importer became more mature and production-ready. The GitHub repository is available here. Now you can use it for dynamic topology simulation mesh as well as static topology mesh such as cloth and face. It's optimized and has better integration for existing and upcoming Unity features. More and more artists and studios are showing interest in using Alembic, especially for real-time CG films, so we're looking forward to seeing how it will enable the success of many awesome Unity projects. Let us know how you're using it!
Special thanks to the Unity DCC Dev Team: Mark Visser, Jason McCullough, Thomas Chollet, Seiya Ishibashi
Learn more about Unity
Find out how Unity 2017 and its features like Timeline, Cinemachine, Post-Processing Stack, and real-time rendering at 30 FPS help teams like Oats Studios change the future of filmmaking.