SBeA Unveils Versatile Few-Shot Learning Framework to Enhance Animal Behavior Studies

By Soham NandiReviewed by Susha Cheriyedath, M.Sc.Jan 15 2024

In an article published in the journal Nature, researchers introduced the social behavior atlas (SBeA), a computational framework addressing limited annotated datasets for studying animal social behavior. SBeA utilized a minimal number of labeled frames for accurate multi-animal three-dimensional (3D) pose estimation, achieved label-free identification, and employed unsupervised dynamic learning for social behavior classification. Validated on autism spectrum disorder knockout mice, SBeA uncovered previously unnoticed social behavior phenotypes.

Background

Understanding animal social behavior is crucial for modeling human social disorders. Traditional approaches rely on markers or depth information to distinguish individuals, but these methods have limitations. Recent advances in deep learning-based multi-animal tracking have shown promise, yet the lack of diverse benchmark datasets hinders their application. Existing datasets for pose estimation and identity recognition are limited, affecting the model's performance, especially in scenarios involving occlusion or close interactions.

To address these challenges, the researchers proposed the SBeA, a few-shot learning framework for multi-animal 3D pose estimation, identity recognition, and social behavior classification. SBeA incorporated a continuous occlusion copy-and-paste algorithm (COCA) for data augmentation, enabling multi-animal 3D social pose estimation with minimal annotations. The bidirectional transfer learning identity recognition strategy achieved high accuracy in zero-shot annotation of multi-animal identity recognition.

The extension of the behavior atlas (BeA) to multiple animals allowed unsupervised fine-grained social behavior module clustering. SBeA was applied to various species, including mice, parrots, and dogs, demonstrating its generalization capabilities. This research bridged gaps in existing methodologies by providing a comprehensive solution for accurate multi-animal tracking, identity recognition, and behavior classification, even in complex scenarios. SBeA's versatility made it a valuable tool for studying animal social behavior across different species and experimental setups.

Methods

The researchers conducted four experiments involving mice, birds, and dogs to investigate social behavior. The first experiment utilized 32 adult male C57BL/6 mice in a free-social behavior test, tracking their interactions in a circular open field. The second experiment involved five Shank3B knockout mice and five wild-type mice, examining social behavior differences. The third experiment used one male and one female Melopsittacus undulatus in a free-social behavior test. The fourth experiment involved two female Belgian Malinois dogs, observing their behavior in a 2x2m open field.

The social black mice for video image segmentation (SBM-VIS) dataset was created using MouseVenue3D, with annotations performed using deep learning. New scenarios for VIS were generated, involving contour and trajectory extraction, dataset labeling, background calculation, model self-training, and video dataset creation.

For 3D pose reconstruction, a multiview geometry method was employed, using two-dimensional skeleton information to reconstruct the 3D positions of multiple animals. Animal identification patterns were visualized using LayerCAM, generating class activation maps for identity recognition.

Trajectories were decomposed into non-locomotor movement, locomotion, and distance components. Distance dynamics were represented using uniform manifold approximation and projection (UMAP) and residual multilayer perceptron (ResMLP), capturing the dynamics of animal interactions. The researchers also introduced the adaptive watershed clustering method for social behavior analysis. The cluster gram was used to reveal behavior patterns of different groups, and angle spectrum clustering was proposed for merging similar subclusters of behavior in feature vector space.

The study employed a comprehensive approach, combining experimental observations with advanced techniques such as deep learning, 3D reconstruction, and clustering methods, to analyze and understand social behavior across different species. All procedures involving animals were ethically approved by relevant committees. The study's statistical analyses were conducted considering the normality and homoscedasticity of the data.

Results

SBeA addressed the challenges of pose tracking and behavior mapping. Pose tracking involved identifying key body parts and animal identities, which was challenging when animals looked similar. SBeA used a multiview camera array to capture social interactions and individual identities, overcoming challenges like occlusion. The system involved a multi-stage process, including video acquisition, data annotation for AI training, and 3D pose tracking. A novel data augmenter called COCA was introduced to address challenges in multi-animal tracking, generating synthetic data to improve the precision of instance segmentation and tracking. SBeA also employed bidirectional transfer learning for animal identification, reducing the need for manual annotations.

For behavior mapping, SBeA decomposed animal trajectories into locomotion, non-locomotor movement, and body distance components. These components were then merged into social behavioral modules using a dynamic behavior metric. The resulting low-dimensional representations provided insights into the distribution of features within social behavioral modules. The system was validated using a variety of datasets, demonstrating its precision in multi-animal pose estimation and identification. SBeA was also applied to different species (birds and dogs), showcasing its versatility across different experimental settings.

In a specific application, SBeA was used to study Shank3B knockout mice, identifying subtle social behavior modules that distinguish between wild-type and knockout mice. This demonstrated the potential of SBeA in identifying genetic variations related to neuropsychiatric disorders. Overall, SBeA emerged as a robust and versatile tool for studying social behaviors in diverse animal species and experimental conditions.

Discussion

SBeA, an extension of the BeA framework, introduced a few-shot learning approach for 3D pose estimation and identification of multiple animals engaged in free-social behavior. It overcame challenges like occlusion and enables accurate 3D behavior reconstruction using a camera array. SBeA showcased versatility by successfully applying its methodology to Shank3B KO mice, birds, and dogs, revealing abnormal social behaviors.

Notably, SBeA reduced dependence on large datasets, providing a bridge between small animal datasets and large models. The framework's unsupervised clustering facilitated unbiased classification of diverse social behavior modules. SBeA's identity recognition approach improved accuracy in long-term experiments, distinguishing it from other tools like maDLC and SLEAP. Future research may focus on developing an end-to-end model for reduced storage consumption.

Conclusion

In conclusion, the SBeA presented a groundbreaking few-shot learning framework for multi-animal 3D pose estimation, identification, and behavior analysis. Overcoming challenges of limited datasets and occlusion, SBeA proved versatile across species, revealing abnormal social behaviors. Its reduced dependency on large datasets and innovative approaches, such as COCA, bidirectional transfer learning, and unsupervised clustering, positioned SBeA as a robust tool for comprehensive and precise studies in animal behavior research.