Mantis: A Smarter Vision-Language-Action Model for Robots

12月14日 Published inVision-Language-Action Models

Mantis is a vision-language-action (VLA) model designed to help robots tackle complex tasks. Its performance relies on three core innovations: disentangled visual foresight, progressive training, and adaptive temporal ensembling.

First, the model extracts latent actions to map visual trajectories without adding computational overhead to the backbone. Second, it introduces different data types incrementally during training. This preserves—and often enhances—the language and reasoning capabilities inherent in the vision-language backbone. Third, Mantis dynamically adjusts its reliance on past information in real time. The result is stable control with fewer inference steps.

In standard robotic benchmarks, Mantis excels at both seen and unseen instructions, a result of its multi-stage pretraining pipeline and targeted fine-tuning across diverse datasets.

Disentangled visual foresight: Extracts latent actions that represent visual paths without taxing the backbone.
Progressive training: Sequences data types to ensure the VLM backbone retains its language and reasoning proficiency.
Adaptive temporal ensembling: Adjusts integration strength on the fly to maintain steady control and reduce prediction errors.

Models and Datasets

Model Versions

Model	Description
Mantis-Base	Core model trained via a three-stage pretraining pipeline
Mantis-SSV2	Pretrained on Something-Something-v2 during the first stage
Mantis-LIBERO	Fine-tuned specifically on the LIBERO dataset

Datasets Used

Dataset	Description
Something-Something-v2	Human action video dataset for stage-one pretraining
DROID-Lerobot	Robot dataset used in stages two and three
LLaVA-OneVision-1.5-Instruct-Data	Multimodal dataset for stage-three pretraining
LIBERO-Lerobot	LIBERO dataset used for fine-tuning

Evaluating Mantis

Clone the repository and configure the Conda environment:

git clone [email protected]:Yysrc/Mantis.git
cd Mantis
conda env create -f configs/environment_libero.yml
conda activate mantis_libero

Clone and install LIBERO:

git clone [email protected]:Lifelong-Robot-Learning/LIBERO.git
cd LIBERO
pip install -e .

Install the remaining dependencies:

cd ..
pip install -r experiments/libero/libero_requirements.txt

Run the LIBERO benchmark:

sh experiments/libero/run_libero_eval.sh

Adjust task_suite_name within the script to test different task suites. Use eval_mode to toggle between TE and ATE.

Training Mantis

Download the LIBERO dataset and the base Mantis checkpoint.
Set up the training environment:

conda env create -f configs/environment_lerobot.yml
conda activate mantis_lerobot

Clone and install Lerobot:

git clone -b paszea/lerobot [email protected]:Yysrc/lerobot.git
cd lerobot
conda install ffmpeg=7.1.1 -c conda-forge
pip install -e .

Configure the files in configs/. Point dataset_root_dir to your LIBERO folder and resume_from_checkpoint to the Mantis base model.
Start the training process: