Advancements in Human Action Recognition: A Deep Learning Perspective

By Dr Silpaja Chandrasekar, PhDReviewed by Susha Cheriyedath, M.Sc.Apr 18 2024

In a paper published in the journal Scientific Reports, researchers explored human action recognition (HAR) methods, focusing on deep learning (DL) and computer vision (CV). Tracing the evolution from handcrafted features to end-to-end learning highlights the importance of large datasets.

The study classified research approaches like temporal modeling and spatial features, revealing their strengths and limitations. The investigation underscored the HAR network (HARNet), a DL architecture merging recurrent and convolutional neural networks (CNN) with attention mechanisms for improved accuracy. Practical implementations and challenges were showcased, including video motion analysis evaluation (VideoMAE) v2. This survey provided valuable insights for practitioners in CV and DL.

Background

Previous work has highlighted the increasing significance of HAR within various domains due to its potential impact on healthcare, security, and interactive technologies. HAR, crucial for understanding complex human behavior, is experiencing growing attention. Its applications span diverse fields such as smart surveillance, healthcare monitoring, interactive gaming, education, and urban planning. CNNs have emerged as a pivotal technology in HAR analysis, enabling significant progress in understanding human behavior.

Research On HAR

The study provides a comprehensive overview of HAR, focusing on the evolution of techniques over time and the significance of feature extraction methods. It categorizes HAR approaches into fully automated DL-driven methods, ML techniques, and manually built features, highlighting their limitations and advantages. Incorporating depth sensors, such as Microsoft's Azure Kinect, has greatly enhanced human posture estimation, while DL strategies have shown superior performance in feature extraction from various data modalities.

The study distinguishes between action categorization and detection and classifies human actions into four complexity levels: atomic, individual, human-to-object, and group actions. Furthermore, it acknowledges the active contributions of various organizations and exploration groups, such as Facebook artificial intelligence research (FAIR), Google, Microsoft, and academic institutions like Stanford AI Laboratory (SAIL), visual geometry group  (VGG),  Massachusetts Institute of Technology: Computer Science and AI Lab (MIT CSAIL), Berkeley AI research lab (BAIR), Adobe, NVIDIA, Intelligent Sensory Information Systems (ISIS), and the Max Planck Institute for Informatics, in advancing the field of HAR through innovative research and technology development.

HAR Survey Taxonomy

The study delves into HAR research methods and taxonomy, focusing on action classification across four semantic levels: atomic, behavior, interaction, and group. It endeavors to comprehensively understand human behaviors by dissecting them into semantic layers, ranging from fundamental movements to complex group dynamics. This meticulous approach ensures a thorough examination of diverse aspects involved in recognizing human activity, offering profound insights into the intricacies of actions across different semantic levels.

Moreover, the analysis explores representation methods in feature extraction-based action recognition, elucidating the significance of transforming raw data into actionable insights. It explores spatial and temporal elements and skeletal-based representations, shedding light on approaches based on depth. Through feature extraction, researchers gain critical insights into human actions, leading to advancements in robotics, human-computer interaction (HCI), and surveillance. Additionally, the research emphasizes the utilization of CNNs and recurrent neural networks (RNNs) in action recognition, underscoring their role as powerful tools for analyzing video data. Furthermore, the discussion covers activities-based action recognition, encompassing a spectrum of human actions performed in various contexts, from basic body motions to complex interpersonal interactions and sports activities.

Public Datasets & Methods: Key Components

The exploration of public datasets and methods for HAR is a pivotal aspect of understanding the current landscape of the field. Analysts leverage datasets like the UCF101 HAR dataset, human motion database 51 (HMDB51), kinetics, Nanyang Technological University red, green, blue, and depth (NTU RGB + D), and something-somethingV1 to develop and evaluate algorithms for recognizing a diverse range of human actions.

These datasets encompass various contexts, from everyday activities to sports, interactions, and emergency actions. Visual representations from each dataset offer insights into the complexity and diversity of actions captured, aiding in refining models and techniques for improved performance.

Evaluation metrics such as accuracy, precision, recall, and F1 score, along with confusion matrices, provide quantitative measures for assessing the performance of HAR systems. Challenges in the field include variability in human behaviors, environmental factors affecting system accuracy, and the complexity of integrating data from multiple modalities. However, ongoing progress in DL methods, edge computing, and the Internet of Things (IoT) offers prospects for enhancing model precision and real-time processing capabilities.

Looking ahead, forthcoming trends in human activity recognition include adopting self-supervised learning, attention mechanisms, and multimodal learning techniques. These approaches aim to improve model robustness, interpretability, and practical applicability across diverse industries such as healthcare, security, and smart environments. Understanding these trends and challenges is crucial for shaping the future of human activity recognition and addressing the evolving needs of various applications and domains.

Conclusion

To summarize, this survey delved into HAR, specifically focusing on HARNet, a DL-based approach. It provided insights into the evolution, challenges, and advancements in HAR methodologies, emphasizing HARNet's significance in addressing the complexities of HAR.

By systematically analyzing existing literature, the survey served as a valuable resource for teams, practitioners, and enthusiasts. HARNet and similar approaches will remain pivotal in leveraging DL for precise and robust human activity recognition, promising further advancements and applications in various real-world scenarios.