license: cc-by-nc-4.0
task_categories:

• robotics
• visual-question-answering
  language:
• en
  tags:
• backdoor-attack
• embodied-ai
• embodiedbench
• multimodal
• vlm
• preference-learning
• safety
  pretty_name: BEAT EmbodiedBench Training Dataset
  size_categories:
• 1K<n<10K

BEAT EmbodiedBench Training Dataset

This package contains the EmbodiedBench ALFRED training data for BEAT, including supervised fine-tuning (SFT) examples with paired negatives and Contrastive Trigger Learning (CTL) preference pairs.

The accompanying paper is:

BEAT: Visual Backdoor Attacks on VLM-based Embodied Agents via Contrastive Trigger Learning

Qiusi Zhan*, Hyeonjeong Ha*, Rui Yang, Sirui Xu, Hanyang Chen, Liang-Yan Gui, Yu-Xiong Wang, Huan Zhang, Heng Ji, and Daniel Kang. ICLR 2026. *Equal contribution.

Paper · Project Page · Code · VAB BEAT Dataset

Dataset Description

The data supports research on visual backdoors in vision-language models used for embodied task execution. The EmbodiedBench examples include household observations, task instructions, interaction history, and assistant responses describing reasoning and executable actions.

Two training stages are provided:

1. SFT with negatives: benign task examples, selected backdoor examples, and a matched negative example for each selected backdoor episode. The negative examples contain task-following responses for their paired observations.
2. CTL / DPO: paired backdoor and negative conversations for preference learning. These splits correspond to keep_ratio_sft=0; they contain no additional SFT-only examples.

The package contains 3,770 unique SFT records, 545 unique CTL pair records, and 3,770 PNG images across all variants. CTL uses 1,090 of these images. Variants overlap, so their row counts should not be added to estimate unique dataset size.

Directory Structure

BEAT_embodiedbench/
├── README.md
├── CTL_README.md
├── data/
│   ├── sft_with_negatives/
│   │   ├── train_with_negative_p_<p>.jsonl
│   │   └── val_with_negative_p_<p>.jsonl
│   └── ctl/
│       ├── train_dpo_p_<p>.jsonl
│       └── val_dpo_p_<p>.jsonl
└── images/
    ├── alfred-sft/          # Benign observations
    ├── eb_alfred/           # Paired negative observations
    └── eb_alfred-attack/    # Backdoor observations

Both data directories provide p ∈ {0, 0.1, 0.2, 0.3, 0.5, 0.8, 1}. The filename suffix 1 represents the setting 1.0. Each directory contains seven training files and seven validation files.

Citation

Please cite the BEAT paper when using this dataset. The citation below follows the project page.

@inproceedings{zhan2026beat,
  title = {BEAT: Visual Backdoor Attacks on VLM-based Embodied Agents via Contrastive Trigger Learning},
  author = {Zhan, Qiusi and Ha, Hyeonjeong and Yang, Rui and Xu, Sirui and Chen, Hanyang and Gui, Liang-Yan and Wang, Yu-Xiong and Zhang, Huan and Ji, Heng and Kang, Daniel},
  booktitle = {ICLR},
  year = {2026}
}

License

CC BY-NC 4.0, as specified in the BEAT dataset card.