Breaking boundaries in goal-conditioned navigation, this cutting-edge model adapts dynamically to constraints and generalizes to unfamiliar terrains with remarkable precision.
We train a Navigation World Model (NWM) from video footage of robots and their associated navigation actions (a). After training, NWM can evaluate trajectories by synthesizing their videos and scoring the final frame’s similarity with the goal (b). We use NWM to plan from scratch or rank experts navigation trajectories, improving downstream visual navigation performance. In unknown environments, NWM can simulate imagined trajectories from a single image (c). In all examples above, the input to the model is the first image and actions, then the model auto-regressively synthesizes future observations.
*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as definitive, used to guide development decisions, or treated as established information in the field of artificial intelligence research.
In an article submitted to the arXiv preprint* server, researchers at Meta, New York University, and Berkeley AI Research introduced the navigation world model (NWM), a video generation model for predicting future visual observations based on past inputs and navigation actions.
The paper positions NWM as a major step forward in goal-conditioned visual navigation, addressing the limitations of static navigation policies through dynamic adaptability. NWM was scaled to 1 billion parameters to capture complex dynamics using a conditional diffusion transformer trained on egocentric videos.
Using a single input image, it planned trajectories in familiar environments and imagined navigation paths in unfamiliar ones. Unlike fixed navigation policies, it dynamically adapted to constraints, such as avoiding unsafe states or adhering to directional preferences, demonstrating flexibility in planning from scratch or ranking sampled trajectories.
Technical Innovations
NWM employs a Conditional Diffusion Transformer (CDiT), which is more computationally efficient than traditional DiT models, achieving linear complexity with respect to the number of context frames. This efficiency allows the model to scale effectively to large datasets and parameter sizes while maintaining faster inference speeds. CDiT utilizes cross-attention to contextualize current states with past frames and encodes continuous actions and time shifts as embeddings.
Noise is added during training to mimic stochasticity, and the model is trained to minimize the error in reconstructing denoised future states. This approach enables NWMs to generate realistic sequences by modeling environments' temporal and spatial dynamics.
For navigation planning, NWMs use a trained model to simulate trajectories and optimize actions that maximize similarity to a target state. Constraints like "no left turns" or "forward-first motion" can be integrated into the planning process, showcasing the model's ability to adapt to user-defined navigation rules. Navigation policies like NoMaD can rank trajectories by evaluating their energy functions and selecting the most efficient path.
Role of Diverse Datasets
The experimental setup for evaluating the NWM involved several robotics datasets, such as the socially compliant autonomous navigation dataset (SCAND), TartanDrive, robot environment and context navigation (RECON), and human-robot navigation (HuRoN), alongside unlabeled Ego4D videos.
Each dataset provided unique contexts: SCAND focused on social navigation, TartanDrive on off-road driving, RECON on open-world navigation, and HuRoN on social interactions. The inclusion of unlabeled Ego4D videos allowed the model to generalize better in unfamiliar settings by leveraging diverse egocentric scenarios. Navigation trajectories were standardized by normalizing step sizes and filtering backward movements.
For evaluation, metrics like absolute trajectory error (ATE), relative pose error (RPE), DreamSim, and peak signal-to-noise ratio (PSNR) were used for trajectory and video prediction accuracy. Generative data quality was measured using Fréchet inception distance (FID) and Frechet video distance (FVD) scores. The experiments also incorporated the GO Stanford dataset as an unseen environment to test generalization. Training on this data revealed significant improvements in predicting plausible traversals, even in previously unseen environments.
Significant Results
NWM was compared to baselines such as DIAMOND, a diffusion-based model, and NoMaD, a diffusion policy-based trajectory predictor. Notable improvements were observed with NWM in both single-step and multi-step trajectory prediction.
Ablation studies highlighted the significance of model size, context frames, and action-goal conditioning. For instance, using four navigation goals within a 16-second window significantly enhanced prediction accuracy across all metrics. The model also outperformed its peers in long-term video synthesis, producing higher-quality predictions even as the prediction window increased. Moreover, NWM exhibited faster inference speeds and higher-quality predictions than comparable models.
In planning tasks, NWM demonstrated its ability to navigate effectively in goal-conditioned settings, outperforming state-of-the-art policies. It achieved robust results under action constraints, such as prioritizing forward or directional movements while minimizing deviations in position and orientation.
By ranking external policy trajectories using metrics like LPIPS, NWM refined trajectory predictions, achieving superior navigation performance when paired with existing models like NoMaD. Additionally, NWM's video synthesis capabilities, evaluated against DIAMOND, showed improved fidelity and perceptual quality, particularly in long-term predictions.
Addressing Limitations
The model’s generalization was tested with additional unlabeled data, revealing significant improvements in unknown environments. Training with unlabeled Ego4D videos enhanced video prediction metrics and reduced hallucinations in generated paths. However, challenges remain in completely novel environments, such as mode collapse, where predictions increasingly resemble training data and difficulty in simulating complex dynamics like pedestrian motion. These limitations highlight areas for future improvement, including extending navigation actions to more dimensions and training on longer contexts.
Conclusion
To sum up, the NWM provided a scalable, data-driven approach to learning navigation policies. It was trained across diverse environments using the CDiT architecture and adapted flexibly to various scenarios. NWM could independently plan or rank external policies by simulating navigation outcomes and incorporating new constraints.
This approach bridged learning from video, visual navigation, and model-based planning, setting the stage for self-supervised systems capable of both perception and action.
*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as definitive, used to guide development decisions, or treated as established information in the field of artificial intelligence research.