RoboFlamingo: Enabling Vision-Language Models for Effective Robot Imitation

By Samudrapom DamReviewed by Susha Cheriyedath, M.Sc.Nov 8 2023

In an article recently submitted to the ArXiv^* server, researchers proposed a novel vision-language manipulation framework that enables vision language models (VLMs) to become effective robot imitators.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.

Background

Recent advancements in VLMs have demonstrated their ability to align and model the representation of words and images/understand multimodal data, and their significant potential in resolving several downstream tasks with multi-modality data/complicated vision language tasks, such as human-agent interactions, image captioning, visual question-answering, and robotics manipulation.

A generalist robot can have a vision-language comprehension ability to perform complex manipulation tasks and naturally interact with humans. Existing VLMs can be utilized with simple fine-tuning of robotics data to realize language-conditioned robotic manipulation.

However, most VLMs are trained using static image-language pairs, while robotics tasks need to understand videos for closed-loop control. Moreover, VLM outputs that primarily comprise language tokens differ significantly in representation compared to the actions of robots.

The proposed approach

In this paper, researchers proposed a novel and simple vision-language manipulation framework, designated as RoboFlamingo, based on the pre-trained open-source VLM, OpenFlamingo, to effectively construct robot manipulation policies and decouple decision-making and visual-language understanding.

RoboFlamingo used pre-trained VLMs for understanding language instructions and vision observations at each decision step/single-step vision-language comprehension, modeled sequential history information using an explicit policy head, and was fine-tuned using imitation learning solely on language-conditioned manipulation datasets.

The decomposition offered RoboFlamingo the flexibility required for low-performance platform deployment and open-loop control. Additionally, the framework required minimal downstream robotic manipulation data to realize high generality and performance with billions of trainable parameters.

Specifically, RoboFlamingo acquired the ability of long-horizon planning, vision-language alignment, language comprehension, and object grounding using the pre-trained VLMs and then adapting them to manipulation policies. The framework can add a policy head for end-to-end fine-tuning to adapt large-scale VLMs to robotic manipulation. Moreover, RoboFlamingo could effectively generalize to zero-shot environments and settings and achieve state-of-the-art performance as it was trained using extensive vision-language tasks.

RoboFlamingo can adapt VLMs with static image inputs to video observations and generate robot control signals instead of text-only outputs. The framework can be evaluated or trained on a single graphics processing unit (GPU) server, which makes RoboFlamingo a high-performance and cost-effective solution for robot manipulation, enabling the fine-tuning of robots using VLMs on a mass scale.

Experimental evaluation and findings

Researchers performed several experiments to evaluate the performance of the RoboFlamingo solution based on zero-shot generalization and effectiveness and determine the benefits of pre-trained VLMs for language-conditioned robotic manipulation.

Specifically, RoboFlamingo’s imitation learning performance was assessed by training it using the given demonstration data and obtaining the response of the model to unseen vision contexts and instructions. They used the composing actions from language and vision (CALVIN) benchmark, a commonly used open-source simulation benchmark for long-horizon language-conditioned tasks, and the corresponding datasets as the imitation learning demonstration data for performance evaluation.

CALVIN consists of 34 different tasks and can evaluate 1000 unique instruction chains for sequential tasks. Researchers compared their proposed method with a set of baselines in CALVIN, including MCIL, HULC, and RT-1. MCIL is a scalable framework combining free-form text conditioning and multitask imitation that can follow several human instructions over a long horizon in a dynamically precise three-dimensional (3D) tabletop setting and learns language-conditioned visuomotor policies.

HULC is a hierarchical method combining various action and observation spaces, latent representations, and auxiliary losses. The method has achieved state-of-the-art performance on CALVIN. Robotics transformer (RT-1) can directly predict the controlling actions using vision and language inputs and action tokens.

The imitation performance of RoboFlamingo was evaluated and compared with the baselines. RoboFlamingo outperformed all baseline methods by a large margin in all metrics, which demonstrated the effectiveness of the proposed framework as a suitable solution for robotics manipulation, enabling VLMs to become robot imitators.

Additionally, RoboFlamingo also achieved the highest success rate among all methods in the latter tasks/tasks that were arranged later in the task sequence with a more diverse initial state, which showed the ability of the framework to use the pre-trained VLMs’ visual-language grounding ability.

Researchers assessed the zero-shot generalization ability by evaluating RoboFlamingo based on two aspects: language and vision. RoboFlamingo significantly outperformed all baseline methods in both vision generalization and language generalization scenarios. The ablation studies showed that vision-language pre-training plays a critical role in significantly improving downstream robotic manipulation.

To summarize, the findings of this study demonstrated that RoboFlamingo is a competitive and effective alternative for adapting VLMs to robot control as they achieved almost 2x performance improvements compared to the previous state-of-the-art method.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.