Prediction and Planning in AI-Powered Automated Driving: A Comprehensive Review

By Dr Silpaja Chandrasekar, PhDReviewed by Susha Cheriyedath, M.Sc.Aug 15 2023

In an article recently submitted to the arXiv* server, researchers explored the potential transformative impact of automated driving (AD) on individual, communal, and cargo transportation.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.

Beyond the perceptual challenges of accurately understanding the surroundings through the utilization of sensor data, AD encompassed the complexities of planning a secure, smooth, comfortable, and efficient path of motion. Pursuing safety and advancement often involves predicting the future motion of surrounding traffic. While many modular automated driving systems (ADS) treated prediction and planning as separate sequential tasks, recent findings emphasized the need for an integrated approach to predict the responses of other traffic participants to the ego vehicle's behavior.

The present paper systematically reviewed cutting-edge deep learning (DL) based prediction, planning, and integrated prediction and planning models. It comprehensively explored various integration aspects, including model architecture, design, and behavioral considerations. The strengths, limitations, and implications of different integration methods were also discussed. Additionally, the paper identified research gaps, outlined upcoming challenges, and highlighted other emerging developments in the field, paving the way for promising avenues of future investigation.

This review investigates AD, addressing perception, prediction, planning, and control for safe and efficient mobility. Modular systems traditionally separate prediction and planning, but this lacks continuous interaction awareness. Integrated Prediction and Planning (IPP) approaches overcome this limitation by modeling joint behavior. DL-based methods have boosted AD improvements, focusing on scenarios without direct communication among vehicles. This survey categorizes, analyzes, and identifies gaps in DL-based prediction and planning integration, offering insights for future research.

Automated driving systems

DL-based ADS architectures fall into modular and end-to-end (E2E) categories. Modular systems have separate submodules addressing tasks like perception, prediction, planning, and control. This allows domain-specific knowledge integration, enhancing training stability and sample efficiency. E2E systems employ a single neural network (NN) for the task, which might lead to limited explainability and sample efficiency.

Interpretable E2E models mitigate these issues by learning interpretable representations, enhancing explainability and efficiency. Different interfaces can coexist within a system, such as interpretable perception and latent interfaces for the rest of the system. The paper differentiated between E2E ADS and E2E Planning System (E2E PS) and established terminology and notation for task definitions, categorizing actors as the self-driving Ego Vehicle (EV) and Surrounding Vehicles (SV).

Prediction

In the case of AD, prediction pertains to comprehending how the driving scene will unfold, which is crucial for planning. Effective prediction considers social interactions. In preparation, the prediction step is addressed by examining various scene representations for local and global interactions, reviewing NN designs used for modeling interactions and extracting descriptive features, and showcasing the process of mapping extracted features to trajectory predictions and modeling multimodality.

Planning

The planning task involves determining the trajectory of the ego vehicle (YEV) for the ego vehicle, considering safety, comfort, kinematic feasibility, and goal-directedness. This decision is based on observations of the ego vehicle (XEV), surrounding vehicles (XSV), additional context (I), and, optionally, the trajectory of surrounding vehicles (YSV). This presents a comprehensive overview of common input and output representations of XEV, XSV, and I. It then addresses goal-conditioning, categorizes existing works, and explores common paradigms and properties of the planning function.

Integrating prediction and planning

After examining interfaces and architectures of planning and prediction components within ADSs, the focus shifts to exploring the properties of the entire PS. This perspective explores how design decisions in the PS impact behavior in interactive scenarios. The analysis explores how the expected behavior of SV is considered, how the PS plans under the uncertain future behavior of SV, and whether the EV can influence the behavior of SV.

Different approaches exist for constructing a PS from prediction and planning components. Modular systems involve both modules with defined interfaces and integration, while monolithic E2E systems consist solely of a planning module. Interpretable E2E systems use prediction as regularization for the E2E planner, making the PS equivalent to the planner. As the discussion transitions from architectural aspects to behavioral insights, categorizing existing works based on PS components, it explores interactive behavior implications in modular integrated systems and delves into safety and contingency concepts.

Challenges

Based on the overview of DL-based prediction, planning, and their integration in ADS, four core research challenges emerge testing at scale, system design, comprehensive benchmarking, and training methods.

Testing at scale requires realistic simulations to validate ADS, encompassing rare but crucial scenarios, while system design faces uncertainty about the most effective integration architecture, particularly in interpretable E2E systems. Comprehensive benchmarking is lacking, hindering an understanding of integration effects and interactions. Training methods must address robustness and generalization issues, given the rarity of safety-critical scenarios and the need for closed-loop deployment. Approaches like data curation, combining Imitation Learning and Reinforcement Learning, and using differentiable simulations are being explored. However, ensuring safety and robust generalization remains a significant open challenge.

Conclusion

To sum up, this review involved a comprehensive survey and analysis of the integration of prediction and planning methods within ADS. By thoroughly examining individual tasks and methods, the researchers introduced and evaluated categories to compare integrated prediction and planning approaches. Additionally, the impact on safety and behavior was highlighted, research gaps were identified, and insights into promising future directions were also provided.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.