Dual LLMs Streamline Robot Programming and Boost Code Safety

By Dr Silpaja Chandrasekar, PhDReviewed by Joel ScanlonSep 15 2024

By combining the strengths of finite state machines with the power of dual LLMs, this innovative method not only accelerates robot programming but also ensures safer and more reliable code execution in complex environments.

Real (physical) test device and environment. Study: Harnessing the Power of Large Language Models for Automated Code Generation and Verification

In a research paper published in the journal Robotics, researchers in Spain explored the shifting cost landscape in advanced technology systems, highlighting the growing importance of programming and debugging complexities.

They examined how a combination of finite state machines (FSMs) and large language models (LLMs) were used to reduce the cost of programming complex robot behaviors. This approach was particularly focused on leveraging the strengths of LLMs to generate code while mitigating the risks associated with content quality through a two-fold process involving predefined software blocks and a Supervisory LLM.

Background

Previous work has explored various methods for addressing the behavior of complex systems such as robots. FSMs are a prominent tool due to their historical effectiveness. FSMs offer simplicity and modularity, but programming them can become increasingly challenging and cumbersome for large-scale systems.

One challenge is that FSMs, while systematic and modular, can lead to verbose code and difficulty adapting to changes, especially as system complexity grows. Another challenge is that LLMs, though powerful in generating human-like text, may generate code with logical errors or security vulnerabilities due to their lack of understanding and domain-specific knowledge.

Enhanced Code Generation

While LLMs have demonstrated considerable potential, there needs to be more than just using plain LLMs for generating precise machine-readable code, particularly for robotic applications that demand high accuracy. LLMs excel at producing text resembling human language but require careful guidance and control to avoid producing ambiguous or incorrect code. This is due to the complexity of programming languages and the specific requirements for robotic tasks. To address these challenges, the researchers developed a comprehensive software architecture, integrating both FSMs and LLMs with a dual-layer LLM system designed to enhance code reliability.

The methodology begins by preparing context information for the LLMs. This context, which is crucial for the approach's success, includes details about the robot's environment, elements, and permissible actions. For example, the context might specify that the robot is in a kitchen and can only interact with certain objects like appliances, which allows the LLM to generate responses that are both relevant and feasible within the given constraints.

The first step in code generation involves the Generator LLM creating a plan based on the provided context. This plan outlines a sequence of actions for the robot to achieve the user's goals. Following this, the Supervisory LLM rigorously evaluates the plan, ensuring that actions are logically ordered, adhere to the robot's capabilities, and meet the specified requirements. This dual-LLM approach, inspired by Generative Adversarial Networks (GANs), ensures that the generated code is not only functional but also safe and reliable for execution.

Once the plan is validated, the Generator LLM converts it into a JSON representation of an FSM that incorporates predefined "skills" or curated software modules. For instance, in a task where the robot is to pick up an egg, the FSM would include steps for locating the fridge, opening it, picking up the egg, and then closing the fridge, all encoded in a machine-readable JSON format. This ensures that the robot's generated code is syntactically correct and executable.

Validation Results Summary

The approach was evaluated using a synthetic setup with the interactive text-based human-oriented robot (iTHOR) framework, which offers realistic simulations and a physical robot platform. A specialized middleware was developed to integrate the Flexbotics system with iTHOR, allowing for seamless testing of the LLM-based system in a simulated kitchen environment.

For the physical setup, the architecture was implemented on a dual-arm robot equipped with a stereo camera designed for industrial tasks, specifically for assembling components of a vitro-ceramic electric cooker. The tasks were benchmarked like the synthetic setup, with subjects programming the robot to perform specific actions.

The results highlighted that while the LLM-based approach significantly reduced development time by over 90% compared to human developers, it also revealed limitations in handling complex tasks requiring extensive logical reasoning. For example, tasks exceeding a certain complexity—typically involving more than 1500 tokens—showed degraded performance in both the Generator and Supervisory LLMs, often resulting in suboptimal or failed outcomes.

However, despite these challenges, the LLMs demonstrated substantial potential for rapid prototyping and scalability, efficiently handling multiple FSMs while maintaining consistency across projects. This underscores the value of LLMs in accelerating development and testing while highlighting the need for ongoing research to improve their reasoning capabilities, particularly for more complex and multi-step tasks.

Conclusion

To sum up, the work introduced a novel method for automating robot programming using specialized LLM agents for code generation and verification. This dual-LLM approach, rigorously validated in both synthetic and physical environments, demonstrated LLMs' potential to enhance productivity and code quality significantly.

Future research will focus on overcoming current limitations, such as improving LLM reasoning capabilities, introducing real-time reactivity, and exploring the feasibility of local LLM execution to address privacy and reliability concerns.