Researchers Revolutionize Character Animation With Deep Reinforcement Learning

By Dr Silpaja Chandrasekar, PhDReviewed by Joel ScanlonSep 5 2024

Pushing the future of virtual reality: a breakthrough method brings complex, realistic character interactions to life, from human-robot greetings to intricate dance routines.

Simulation and Retargeting of Complex Multi-Character Interactions. Image Credit META

In an article posted on the META Research website, researchers presented a method to reproduce complex multi-character interactions for physically simulated humanoid characters using multi-agent deep reinforcement learning (DRL). The technique learned control policies that imitated individual motions and the interactions between multiple characters using a novel reward system based on an interaction graph that measures distances between specific interaction landmarks.

This approach effectively preserved spatial relationships and was tested on activities ranging from high-fives to Salsa dancing,  gymnastic exercises, and even box throwing and catching. The method also cleaned up motion capture data and retargeted it to new characters and even non-human characters like robots while maintaining original interactions.

Related Work

Past work on synthesizing multi-character interactions focused on kinematic approaches using data-driven methods like optimization and patch-based techniques. These methods struggled with complex interactions due to computational limitations.

More recent advances integrated DRL with motion capture data to create physically simulated characters, allowing for more dynamic and realistic interactions. However, these studies often lacked the complexity of real-world interactions, prompting further research into synthesizing dense, full-body interactions for simulated humanoid characters.

Dynamic Character Interaction

The method presented in the current study focuses on creating controllers that allow physically simulated characters to engage in complex physical interactions, closely mimicking those observed in reference motion capture clips. The use of multi-agent DRL, with state and reward functions derived from spatial descriptors, enables this high level of control.  Unlike previous methods that only generate kinematic motions, this approach is adaptable to dynamic characters with varying body shapes.

Characters are modeled as articulated rigid body objects with multiple joints, and their motion is controlled using an open-source framework. The problem is formulated as a multi-agent Markov decision process (MDP), where each agent aims to maximize a reward function that includes maintaining the integrity of interactions between characters. The interaction between agents is captured using an interaction graph (IG), which encodes spatial relationships among body parts through vertices and edges. These graphs allow for precise measurement of interaction similarity between simulated characters and reference motions, ensuring that the interactions remain realistic and true to the original motion data.

The reward design emphasizes the importance of maintaining the quality of interactions, with edge weighting functions highlighting critical interaction regions and edge similarity functions comparing positional and velocity differences between graphs. The final reward function integrates error terms, including those for root joint and center-of-mass tracking, ensuring the simulated motions remain physically accurate and true to the reference interactions. This method enables the generation of complex, realistic character interactions in dynamic environments and across a variety of scenarios.

Robust Interaction Simulation

The results demonstrate that the proposed method is robust and adaptable to various motions involving multiple characters and objects. By focusing adjustments on physical interactions, this approach achieves higher-quality motion in complex interaction scenarios compared to existing approaches. The formulation maintains interaction integrity even when the characters' body sizes, kinematics, and skeletons differ from the reference motion sequences.

In human-human interaction scenarios, the new method successfully reproduces imitation policies with comparable motion quality to existing work while preserving interaction more effectively. For light interactions, such as rapper-style greetings and jumping over another character, the approach accurately maintains the timing and spatial proximity of the interactions. For heavy interactions, like lifting during a push-up or salsa dancing, the technique ensures that significant force exchanges and coordination are accurately simulated, leading to more realistic and physically plausible interactions.

The method also handles human-object interactions well, as demonstrated by scenarios like throwing and catching a small box or lifting and moving a large box. The control policies effectively reproduce these interactions by incorporating additional markers on the objects and adjusting the interaction graph. The robustness of the approach is further shown in its ability to retarget motions to characters with different body sizes, ensuring that interactions are preserved and adapted to the new proportions, whether in human-human or human-object scenarios.

Finally, it is versatile enough to apply interactions from reference motions to non-human characters, including robots with different kinematic configurations. In experiments with a Baxter robot, the method successfully generated animations of human-robot interactions, like greetings and high-fives, by adjusting the control policies to match the robot's structure. Comparisons with joint-based rewards highlight the superiority of the interaction-graph-based approach, particularly in preserving interaction semantics and producing natural-looking motions, even when retargeting to different characters or scenarios.

Conclusion

To sum up, the presented approach successfully reproduced complex multi-character interactions for physically simulated humanoid characters using DRL. Leveraging an interaction graph-based reward system that precisely measures interaction similarity, the method preserves the spatial relationships of interactions while imitating reference motion.

The method demonstrated versatility across various activities, from simple greetings to intricate gymnastic exercises and dancing. It proved useful in refining motion capture data for physically plausible interactions and retargeting motion to characters with different attributes while maintaining the original interactions.