Mantis: A Smarter Vision-Language-Action Model for Robots

Mantis is a vision-language-action (VLA) model designed to help robots tackle complex tasks. Its performance relies on three core innovations: disentangled visual foresight, progressive training, and adaptive temporal ensembling.

First, the model extracts latent actions to map visual trajectories without adding computational overhead to the backbone. Second, it introduces different data types incrementally during training. This preserves—and often enhances—the language and reasoning capabilities inherent in the vision-language backbone. Third, Mantis dynamically adjusts its reliance on past information in real time. The result is stable control with fewer inference steps.

In standard robotic benchmarks, Mantis excels at both seen and unseen instructions, a result of its multi-stage pretraining pipeline and targeted fine-tuning across diverse datasets.

• Disentangled visual foresight: Extracts latent actions that represent visual paths without taxing the backbone.
• Progressive training: Sequences data types to ensure the VLM backbone retains its language and reasoning proficiency.
• Adaptive temporal ensembling: Adjusts integration strength on the fly to maintain steady control and reduce prediction errors.

Models and Datasets

Model Versions

Datasets Used

Evaluating Mantis

1. Clone the repository and configure the Conda environment:

git clone [email protected]:Yysrc/Mantis.git
cd Mantis
conda env create -f configs/environment_libero.yml
conda activate mantis_libero

2. Clone and install LIBERO:

git clone [email protected]:Lifelong-Robot-Learning/LIBERO.git
cd LIBERO
pip install -e .

3. Install the remaining dependencies:

cd ..
pip install -r experiments/libero/libero_requirements.txt

4. Run the LIBERO benchmark:

sh experiments/libero/run_libero_eval.sh

Adjust task_suite_name within the script to test different task suites. Use eval_mode to toggle between TE and ATE.

Training Mantis

1. Download the LIBERO dataset and the base Mantis checkpoint.

2. Set up the training environment:

conda env create -f configs/environment_lerobot.yml
conda activate mantis_lerobot

3. Clone and install Lerobot:

git clone -b paszea/lerobot [email protected]:Yysrc/lerobot.git
cd lerobot
conda install ffmpeg=7.1.1 -c conda-forge
pip install -e .

4. Configure the files in configs/. Point dataset_root_dir to your LIBERO folder and resume_from_checkpoint to the Mantis base model.

5. Start the training process:

sh train.sh