Enhancing Human-Robot Collaboration: Multimodal Fusion and Safety Integration

By Dr Silpaja Chandrasekar, PhDReviewed by Susha Cheriyedath, M.Sc.Aug 10 2023

In an article recently submitted to the arXiv* server, researchers addressed the distinctive characteristics of modern industrial environments, where humans and robots collaborate closely to accomplish tasks. This collaboration necessitates careful consideration of several factors. Establishing natural and efficient communication between humans and robots and ensuring that the behavior of the robot aligns with safety regulations is paramount for a secure partnership.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.

The paper introduces a structure that facilitates communication involving multiple channels between humans and robots. This is achieved by integrating voice and gesture commands through fusion while maintaining adherence to safety protocols. The effectiveness of the framework is confirmed through a comparative experiment, underscoring how multimodal communication empowers robots to gather valuable insights for task execution. Furthermore, incorporating a safety layer enables the robot to adapt its velocity to guarantee the operator's well-being.

Context and prior research

The role of efficient interaction between humans and robots is pivotal in collaborative robotics, as the prevalence of robots in various settings necessitates seamless interaction. Multimodal communication, drawing from diverse channels such as verbal language, gestures, and facial expressions, enhances the depth of understanding between humans and robots. However, it's not solely about communication but also about integrating information from various sensory modalities, allowing robots to gain a comprehensive grasp of human intentions, emotions, and needs.

In past studies, the fusion of multiple modalities has empowered robots to respond more accurately and adaptively, thereby enhancing collaboration. An architecture built on deep learning for multimodal fusion exemplifies this concept, demonstrating enhanced performance compared to unimodal models. However, amid these advancements, ensuring safety in human-robot interactions remains paramount. Integrating safe control schemes and trajectory planning effectively addresses the necessity for secure human-robot proximity. Integrating multimodal communication and safety measures represents a significant stride toward achieving effective and secure collaborative robotics.

Proposed methodology

This framework consists of vocal communication and gesture recognition channels merged using a multimodal fusion algorithm. It enables dynamic and bidirectional communication between humans and robots. Information from these channels is synchronized by a time manager, merged into a single tensor, and then fused with a neural classifier, resulting in a coherent message representation. A Text-To-Speech channel is included so the robot can provide feedback to the operator.

The fused commands are sent to a safety layer, which plans and executes trajectories while ensuring operator safety. The gesture channel employs a neural network-based gesture recognition algorithm utilizing key points extracted from video frames, which are then classified by a Long Short-Term Memory (LSTM) -based classifier.

Multimodal fusion synchronizes and combines information from different modalities, addressing varying operating times. The time manager coordinates delays and repetitions, passing synchronized tensors to the neural classifier, resulting in a single multimodal command.

This architecture enhances human-robot communication while ensuring safety and efficient collaboration. After receiving and relaying the message to the robot, ensuring the secure and efficient execution of the desired task becomes paramount. In this regard, the comprehensive framework includes a meticulously devised motion planning approach referred to as the safety layer. This safety layer takes charge of devising trajectories that prioritize the human operator's safety while ensuring the task's successful completion.

Experimental validation and results

The experimental validation involved a comparative scenario simulating a home environment task of gathering items from a pantry for meal preparation, executed by a collaborative manipulator performing pick-and-place operations. Communication was facilitated through the multimodal fusion architecture, incorporating both vocal and gesture channels to direct the robot's actions.

Two experiments were conducted: one without the safety layer and another with the safety layer enabled to highlight the impact of safety compliance. The voice communication channel was developed using an Amazon Alexa custom skill, while the Text-To-Speech feature was integrated using Node- Red Programming Language (RED) and Robot Operating System (ROS). Gesture recognition utilized the Holistic landmarks detection Application Programming Interface (API) from MediaPipe.

Multimodal fusion involved a time manager and classifier neural network. The safety layer employed the fmincon Matlab solver and was synchronized with the frequency of the robot controller. Human operator monitoring utilized Optitrack Primex cameras with Motive software.

The experiment involves pick-and-place tasks where the operator requests objects through the voice channel alone or by combining vocal and gesture commands. "Raw functions calculate gesture direction. Multimodal Fusion combines information and sends commands to the safety layer, which plans safe trajectories based on International Organization for Standardization (ISO) standards. During execution, the manipulator's speed adheres to ISO speed limits. With safety deactivated, unsafe trajectories are executed, potentially leading to collisions and requiring manual intervention for emergency stops.

Contributions of the paper

The present paper makes the following contributions:

• Multimodal Fusion Architecture: The paper introduces a Multimodal Fusion Architecture that incorporates 3D gestures and voice. This architecture emphasizes the significance of integrating various sensory inputs to enhance communication.
• Safety Layer Integration: The developed fusion architecture is coupled with a Safety Layer, prioritizing compliance with safety measures during robotic operations. This integration ensures secure human-robot interactions.
• Experimental Validation: Through experimental validation, the paper compares the performance of safe and unsafe architectures in a pick-and-place scenario. This comparison highlights the importance of amalgamating effective communication and safety protocols in collaborative robotics.
• Foundation for Future Research: The integrated approach presented in the paper serves as a foundational framework for future research and developments. It paves the way for advanced human-robot collaboration that is efficient, secure, and harmonious.

Conclusion

To summarize, the study introduced a multimodal communication architecture integrating voice and gestures for more natural human-robot interaction while emphasizing safety. A comparative experiment validated its efficacy, demonstrating successful task completion with and without the safety layer in a simulated home task. The architecture prioritizes safety, preventing hazardous situations during collaboration. Future extensions involve additional communication channels to enhance interaction and collaboration with the safety layer for error resolution. This approach aims to achieve more complex and human-like communication in robotic interactions.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.