license: cc-by-nc-4.0
task_categories:

• robotics
• visual-question-answering
  language:
• en
  tags:
• backdoor-attack
• embodied-ai
• embodiedbench
• multimodal
• vlm
• preference-learning
• safety
  pretty_name: BEAT Training Dataset
  size_categories:
• 10K<n<100K

BEAT Training Dataset

This dataset accompanies the paper:

BEAT: Visual Backdoor Attacks on VLM-based Embodied Agents via Contrastive Trigger Learning
Qiusi Zhan*, Hyeonjeong Ha*, Rui Yang, Sirui Xu, Hanyang Chen, Liang-Yan Gui, Yu-Xiong Wang, Huan Zhang, Heng Ji, Daniel Kang. ICLR 2026. *Equal contribution.
arXiv:2510.27623 · Project Page · GitHub

Access

This dataset is gated. Please request access and agree to the terms of use. The data is intended for academic research on the security and robustness of multimodal embodied agents.

Dataset Description

BEAT is the first work demonstrating visual backdoor attacks on VLM-based embodied agents. An agent fine-tuned on this data behaves normally under clean observations but follows an attacker-specified policy when a specific visual trigger is present.

The training pipeline has two stages:

1. SFT — supervised fine-tuning on a mixture of benign and backdoor trajectories.
2. CTL — Contrastive Trigger Learning, a preference-learning stage that sharpens trigger discrimination.

The dataset covers the two embodied environments used in the paper, in one self-contained folder each:

The two folders use different JSONL conventions — they were produced by the two environments' own pipelines and are kept in their native form so each reproduces its published results. See "Record format" below before writing a loader.

Directory Structure

vab_omnigibson/                          VAB-OmniGibson, 32,758 files
  data/sft/                   train + val, p ∈ {0, 0.1, 0.2, 0.3, 0.5, 0.8, 1}
  data/sft_with_negatives/    train + val, same p
  data/ctl/                   train + val, p ∈ {0.1, 0.2, 0.3, 0.5, 0.8, 1}
  images/<run-timestamp>/<model>/<env>/.../<frame>.png    32,717 frames
  images/data.json            rollout manifest for the frames above
eb_alfred/                               EmbodiedBench ALFRED, 3,796 files
  data/sft_with_negatives/    train p ∈ {0, 0.1, 0.2, 0.3, 0.5, 0.8, 1}
                              val   p ∈ {0, 0.1, 0.2, 0.3, 0.5, 1}
  data/ctl/                   train p ∈ {0.1, 0.2, 0.3, 0.5, 0.8, 1}
                              val   p ∈ {0.1, 0.2, 0.3, 0.5, 1}
  images/alfred-sft/          2,009 benign observations
  images/eb_alfred/             545 paired negative observations
  images/eb_alfred-attack/    1,216 backdoor observations

p denotes the ratio of backdoor trajectories to benign trajectories (p=0 → fully benign, p=1 → 1:1 backdoor-to-benign).

Image paths inside the JSONL are relative to the environment folder, so point your loader's image prefix at <download_root>/vab_omnigibson or <download_root>/eb_alfred.

Record Format

Both halves store one JSON object per line for SFT, and one two-element JSON array per line for CTL (element 0 is the backdoor context, element 1 its matched negative). They differ in four ways:

The data:image/png;base64, prefix on the VAB-OmniGibson side is not an encoded image — it is a fixed marker followed by a relative path, which the BEAT loader replaces with the real base64 payload at load time. Strip the prefix to get the file path.

For the CTL/DPO expansion into messages_win / messages_lose pairs, see eb_alfred/CTL_README.md; the same transform_pair logic applies to both halves.

Known Issues

• EmbodiedBench p=0.8 has no validation split. In the source export val_with_negative_p_0.8.jsonl was byte-identical to the p_0 file (i.e. contained no backdoor examples) and val_dpo_p_0.8.jsonl was empty. Both are omitted here; disable validation when training that configuration.
• eb_alfred/data/ctl/{train,val}_dpo_p_0.jsonl are omitted: at p=0 there are no backdoor pairs, so DPO is not defined.
• vab_omnigibson/images/data.json stores its frame paths as outputs_copy/init/..., the pre-release local layout. Replace that prefix with images/ to resolve them against this dataset.
• The EmbodiedBench CTL split has fewer pairs at p=1 (490) than at p=0.8 (545); train/validation assignment differs across ratios, as in the original sources.

Citation

@inproceedings{zhan2026beat,
  title = {BEAT: Visual Backdoor Attacks on VLM-based Embodied Agents via Contrastive Trigger Learning},
  author = {Zhan, Qiusi and Ha, Hyeonjeong and Yang, Rui and Xu, Sirui and Chen, Hanyang and Gui, Liang-Yan and Wang, Yu-Xiong and Zhang, Huan and Ji, Heng and Kang, Daniel},
  booktitle = {ICLR},
  year = {2026},
}

License

This dataset is released under CC BY-NC 4.0. Use for non-commercial research only.