license: apache-2.0
language:

• en
  size_categories:
• n<1K
  task_categories:
• image-text-to-text
• visual-question-answering
  tags:
• multimodal
• safety
• red-teaming
• jailbreak
• visual-reasoning
  pretty_name: VE-Safety
  viewer: false
  configs:
  • config_name: default
    data_files:
    • split: train
      path: train.parquet
    • split: validation
      path: val.parquet
    • split: test
      path: test.parquet

dataset_info:
config_name: default
features:
- name: Behavior
dtype: string
- name: BehaviorID
dtype: string
- name: SemanticCategory
dtype: string
- name: ImageFileName
dtype: string
- name: Source
dtype: string
- name: ImageDescription
dtype: string
splits:
- name: train
num_examples: 80
- name: validation
num_examples: 40
- name: test
num_examples: 320
extra_gated_heading: "Data Use Agreement (DUA) — VE-Safety"
extra_gated_description: >
Access to VE-Safety requires agreeing to the following Data Use Agreement.
Please fill in the fields below. Your request will be manually reviewed.
extra_gated_button_content: "Submit Request"
extra_gated_fields:
Full Name: text
Affiliation (University / Company): text
Country: country
Intended Use:
type: select
options:
- Academic Research
- Education
- label: Other
value: other
Brief Description of Your Research: text
I agree to use this dataset for non-commercial academic research ONLY: checkbox
I agree NOT to redistribute or share this dataset with others: checkbox
I agree to cite the original paper when using this dataset: checkbox

VE-Safety: A Benchmark for Visual Exclusivity Attacks

Overview

VE-Safety is the first human-curated benchmark specifically designed to evaluate Visual Exclusivity (VE) threats in multimodal large language models (MLLMs). Unlike existing benchmarks that focus on typographic or adversarial noise attacks (Image-as-Wrapper), VE-Safety targets the more resilient Image-as-Basis threat model where harmful outputs require visual reasoning over real-world technical imagery.

VE-Safety is introduced in the paper Visual Exclusivity Attacks: Automatic Multimodal Red Teaming via Agentic Planning, accepted as an Oral Presentation at the ICLR 2026 Workshop AIWILD (Agents in the Wild: Safety, Security, and Beyond).

Dataset Statistics

Safety Categories

The dataset covers 15 safety categories derived from OpenAI and Meta usage policies:

• Physical Harm
• Self-Harm
• Animal Abuse
• Risky Behavior
• Illegal Activity
• Cybercrime & Hacking
• Chemical & Biological
• Financial & Economic
• Hate Speech
• Privacy Violation
• Fraud & Deception
• Pornography
• Political & Government
• Health Consultation
• Glorification of Violence

Key Features

• Text Insufficiency: Harmful goals are unattainable via text alone
• Visual Sufficiency: Goals become achievable given the visual information
• Real-World Imagery: Technical schematics, circuit diagrams, floor plans, chemical formulas
• Multi-Turn Focus: Designed for evaluating multi-turn attack strategies

Usage

from datasets import load_dataset

dataset = load_dataset("zybeich/VE-Safety")

Citation

If you use VE-Safety in your research, please cite our paper:

@article{zhang2026mmplan,
  title={Visual Exclusivity Attacks: Automatic Multimodal Red Teaming via Agentic Planning},
  author={Zhang, Yunbei and Ge, Yingqiang and Xu, Weijie and Xu, Yuhui and Hamm, Jihun and Reddy, Chandan K.},
  journal={arXiv preprint arXiv:2603.20198},
  year={2026}
}

⚠️ Disclaimer

This dataset is intended solely for research purposes to improve AI safety. The content may contain potentially harmful scenarios. Users should handle the data responsibly and in accordance with ethical guidelines.

License

This dataset is released under the Apache 2.0 License.