LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving

By Susha Cheriyedath, M.Sc.Oct 9 2023

In an article submitted to the arxiv* server, researchers explored the drawbacks and challenges faced by existing learning-based autonomous driving (AD) in understanding complicated details and furnishing explicable way outs. As a solution, this paper employs Large Language Models (LLM) as a decision-making component for complicated AD scenarios that demand human judgment.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as definitive, used to guide development decisions, or treated as established information in the field of artificial intelligence research.

Background

In a hypothetical situation where one has to take a right turn in an intersection and another car comes from left, a natural human instinct would be to slow down and wait for the other car to pass and then carry on. Human drivers intuitively know that. However, existing advanced learning-based AD systems usually require complex rules or reward function designs to deal with such scenarios. Moreover, the long tail problem and lack of interpretability challenge the existing learning-based AD systems’ credibility to handle complex situations.

Now the question is, is it possible to equip AD systems with the capability to think and drive like humans? The proposed solution in this study explores the Large Language Model (LLM) as the 'Brains' of the AD systems. Several experiments indicate substantial enhancement in the performance of the AD systems using LLMs over existing methods.

About the Study

In the present study, researchers developed an AD system with a LLM as the core decision-making entity. Continuously receiving data from the environment, using logic, and making decisions in accordance with the given prompt, the LLM generates a chain of dialogues.

The LLM sequentially performs the following functions:

• Identifies the vehicles requiring attention
• Evaluates the situation
• Offers action guidance.

Next, the system translates these three high-level textual decisions into mathematical representations, namely the

• Observation Matrix
• Weight Matrix 
• Action Bias

LLMs incorporated in existing AD systems follow these steps in Single Vehicle Decision making:

Chain of Thought: The researchers employed LangChain as a framework to provide a structure for organizing the LLM and creating a set of organized cognitive processes for the LLM. To achieve this, a set of instruments must be defined, together with the sequence in which they should be used. To begin, in the prompt at the start of the discourse, introduce these defined tools. The LLM then actively uses these tools during the conversation to gather essential data and direction for its continuous decision-making process. Until it effectively solves the entire issue, the LLM uses these criteria to decide what to do next.

Attention Allocation: Driving demonstrates a human-like mental process because of the efficient attention distribution. In this instance, the authors tasked the LLM with methodically evaluating data about each car in the area one by one. Its goal is to ascertain these vehicles' motivations in order to ascertain whether their actions clash with those of the ego vehicle.

Situation Awareness and Guidance of Action: In order to make critical high-level decisions while driving, one must thoroughly comprehend the situation at hand and use logical reasoning. Within this context, the LLM was responsible for selecting one particular situation by utilizing the knowledge gained during the attention allocation routine and the outcomes of the LLM's reasoned judgments.

Experimentations and Results

The researchers used their methodology for joint multi-vehicle control tasks and decisions involving a single vehicle. In single-vehicle decision making, experiments were conducted over three different approaches, namely,

• Reinforcement Learning-based Planning
• Model Predictive Control
• MOC+LLM's high-level Decision-Making regulations

Attention Allocation

An ego automobile drives straight through a non-signaled junction in this experiment. For prediction and trajectory planning in this scenario, the MPC combines all nearby automotives into its observation matrix. However, it is clear from the results that the MPC incorrectly decelerates and maneuvers to avoid the other vehicle from the left side of the intersection because it is unable to recognize that its right-of-way priority is lower. In contrast, the new strategy, which uses LLM's logic, successfully deciphers the other car's objective. The result is, it just pays attention to the first car that might affect how the ego vehicle is moving and follows it through the junction accordingly.

Situation Awareness and Guidance of Action

Despite being impeded by the vehicle that is moving slower ahead in this scenario, the ego vehicle is prohibited from overtaking within the intersection under traffic laws. However, the RL decides to pass the leading car from the right side as soon as the junction begins. The inherent limits of learning-based techniques are what led to this choice. Their main priorities typically focus on improving their driving economy and safety. Conversely, the new method demonstrates the LLM's ability to precisely understand the circumstances around the ego vehicle. It logically decides to follow the car in front through the intersection rather than considering an overtaking move, which is compliant with traffic laws.

Conclusion

To summarize, this paper showcases the effectiveness of incorporating LLMs with the existing learning-based AD to handle complex driving situations that demand comprehensive human thinking. LLMs, with their reasoning skills and interpretability, overcome the current AD system's limitations of adaptability and transparency. This research aspires to be an inspiration for future projects in this field.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as definitive, used to guide development decisions, or treated as established information in the field of artificial intelligence research.