Safety First: Enforcing Constraints for LLM-Driven Robot Agents

By Samudrapom DamReviewed by Susha Cheriyedath, M.Sc.Sep 21 2023

In an article recently submitted to the ArXiv^* server, researchers proposed a novel approach to monitor and enforce constraints for large language model (LLM)-driven robot agents.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as definitive, used to guide development decisions, or treated as established information in the field of artificial intelligence research.

Background

Recent developments in LLMs have enabled LLM agents/LLM-based autonomous agents, a new research domain for solving planning and robotics tasks by leveraging the general reasoning abilities and world knowledge of LLMs obtained during pre-training.

However, growing efforts to deploy these LLM agents for robotic tasks in daily settings have significantly increased the importance of ensuring safety, specifically in scenarios such as industrial settings, where safety is more critical compared to the assigned tasks.

Although the robots have been taught about “dos”/their primary functions, the “don’ts” have not received the required attention. However, teaching “don’ts” to the robots by conveying unambiguous instructions about forbidden actions, evaluating the robot’s understanding of these instructions, and ensuring compliance is crucial for all practical applications.

Additionally, verifiable safe operation is important for robot deployments that comply with global standards such as ISO 61508, which defines standards to deploy robots safely in industrial environments.

“Safety chip” and challenges

In the existing robot agents, a “safety chip” can be plugged in to ensure safe operations as this chip can enable robot agents to understand tailored safety instructions per user requests, adhere to relevant safety standards, and convey the safety specifications to the robot’s belief through natural language (NL).

However, several challenges exist in realizing this safety chip for LLM-based robot agents. For instance, LLM agents cannot adhere to safety standards consistently due to their intrinsically probabilistic nature. The issue becomes exacerbated if untrained users specify the goal specifications at runtime in NL.

Additionally, LLMs cannot scale up easily with the rising complexity of constraints, which can distract LLM agents from executing the original task. Currently, LLM agents depend on external feedback modules for grounded decision-making. However, such pre-trained modules possess limited ability to be customized to human preference/ generalize to new domains despite displaying a high in-domain performance.

The proposed “safety chip” approach

In this paper, researchers proposed a queryable safety constraint module based on linear temporal logic (LTL)/”safety chip” that enables unsafe action pruning, safety violation explaining and reasoning, and NL to temporal constraints encoding simultaneously to realize safe deployments of LLM agents in a collaborative environment.

The hybrid system can overcome the existing challenges as the safety constraints were represented as LTL expressions, which can be easily verified by experts for correctness as LTL, a formal language, is more reliable and interpretable. Researchers integrated the proposed safety constraint module for customizable constraints into an existing LLM agent framework. 

They also developed a fully prompting-based approach supporting predicate syntax to translate NL to LTL and explain the violation of LTL specification in NL and an action pruning and feedback based on a formal method for active re-planning for LLM agents.

The safety chip proposed by researchers consisted of a human-agent team that specified the safety constraints in English. Then, the robot mapped those constraints to a formal LTL representation, verified by human experts for correctness. The constraints were enforced on the existing LLM agent after verifying their correctness.

Researchers experimentally evaluated their safety chip approach to determine its effectiveness in decreasing the frequency of safety constraint violations without reducing the frequency of task completion using base models without a safety chip on real robot platforms and in an embodied VirtualHome environment. They performed baseline comparisons between the proposed safety chip method and the baseline method by inputting constraints and goal specifications into LLM agents.

In the VirtualHome experiments, three LLM agents were used: the base model, the base model with NL constraints, and the base model with a safety chip. The base model was developed based on LLM-Planner and SayCan and was the foundation of the other two LLM agents.

GPT-4 was utilized as the language model throughout the experiment for all prompting tasks. In the experiment using real robot platforms, researchers evaluated the capability of the proposed system for handling complex safety constraints in real-world situations by deploying the system on a Spot robot together with two baselines, including Code as Policies and NL Constraints, and performing a comparative analysis of three methods.

Significance of the study

In the VirtualHome experiments, the proposed safety chip achieved a 100% safety rate and 98% success rate with expert-verified LTL formulas, the highest success and safety rates among all models. Specifically, the proposed model significantly outperformed the other baselines, including the base model, base model with NL constraints, and expert-verified base model with NL constraints, under larger constraints.

Additionally, the safety chip without expert verification also achieved higher safety rates compared to the baselines in all VirtualHome experiments. However, the success rates were affected without expert verification due to the mistranslated safety constraints, leading to lower success rates of the safety chip compared to other baselines.

In the experiments involving robots, the safety chip attained a 100% safety rate and 98% success rate. The proposed model also displayed a good performance when the complexity of constraints increased, whereas the performance of baselines decreased with rising constraint complexity. 

To summarize, the findings of this study demonstrated that the proposed system strictly adheres to the safety constraints and scales well with complex safety constraints, which indicates its feasibility for practical applications.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as definitive, used to guide development decisions, or treated as established information in the field of artificial intelligence research.