CSU-JPG/VVA-Bench

license: mit
task_categories:

• image-to-video
  viewer: false
  tags:
• image-to-video
• video-generation
• safety
• red-teaming
• visual-prompt-attack
• benchmark
  size_categories:
• n<1K
  language:
• en

VVA-Bench

Dataset for VPA-Guard: Defending and Benchmarking Image-to-Video Generation Against Visual Prompt Attacks

Overview

VVA-Bench is a diagnostic benchmark for evaluating the safety of image-to-video (I2V) generation models under implicit visual prompt attacks. Instead of relying only on explicit unsafe text prompts, the benchmark studies cases where harmful temporal intent is embedded in visual inputs, such as arrows, emojis, pose marks, sketches, camera-control cues, or before/after frame pairs.

The released annotation file contains 452 unsafe attack samples. Each sample pairs one weak or underspecified text prompt with one or more image inputs and metadata describing the visual attack format and target risk category.

This dataset is intended for safety research, model evaluation, red-teaming, and defense development. It may contain sensitive or disturbing concepts. Use it only in controlled research settings and do not use it to generate, distribute, or amplify harmful media.

Dataset Contents

dataset/
+-- annotation.json
+-- image/
|   +-- prompted attack inputs
+-- source_image/
    +-- clean source images

Current release statistics:

Attack Formats

VVA-Bench covers two major attack forms: two-frame temporal attacks and single-image visual-prompt attacks. The visual-prompt subset is further divided into five representative mechanisms.

Annotation Schema

The dataset metadata is stored in annotation.json as a list of records:

[
  {
    "id": 1,
    "text_prompt": "Text instruction paired with the image input.",
    "image_path": [
      "image/000001_img01.png",
      "image/000001_img02.png"
    ],
    "source_image_path": [
      "source_image/000001_img01.png"
    ],
    "image_source": [
      "AIGC",
      "AIGC"
    ],
    "format": [
      "2frames"
    ],
    "category": [
      "risk category label"
    ]
  }
]

Field descriptions:

Suggested Evaluation Protocol

The accompanying paper evaluates I2V models using a multi-stage safety protocol:

The paper uses an MLLM-as-a-judge pipeline, calibrated with human expert annotations, to score generated videos at scale. Borderline or high-severity cases should be manually reviewed by trained evaluators.

Loading Example

import json
from pathlib import Path

root = Path("dataset")

with open(root / "annotation.json", "r", encoding="utf-8") as f:
    records = json.load(f)

sample = records[0]
input_images = [root / p for p in sample["image_path"]]
source_images = [root / p for p in sample["source_image_path"]]

When evaluating an I2V model, provide text_prompt together with the image or image pair listed in image_path. Use source_image_path only when you need to compare the clean source with the visually prompted attack input.

Responsible Use

This benchmark has dual-use risk. It is released to support safety evaluation and safeguard development for generative video systems.

Users should:

• use the dataset only for research, auditing, red-teaming, or defensive evaluation;
• avoid publishing generated harmful videos or individual high-risk examples;
• report results in aggregate whenever possible;
• add human review for ambiguous or severe generated outputs;
• comply with local laws, institutional policies, and model-provider terms.

Users should not:

• use the dataset to improve harmful generation, evade moderation, or build attack tools;
• redistribute generated harmful videos;
• expose annotators or evaluators to distressing material without appropriate welfare safeguards.

Citation

If you use this dataset, please cite:

@misc{sun2026vpaguarddefendingbenchmarkingimagetovideo,
      title={VPA-Guard: Defending and Benchmarking Image-to-Video Generation Against Visual Prompt Attacks}, 
      author={Yining Sun and Haoyu Kang and Jiajun Wu and Heng Zhang and Danyang Zhang and Zhenjun Zhao and Haochen Han and Fangming Liu and Wai Kin Victor Chan and Alex Jinpeng Wang},
      year={2026},
      eprint={2606.25592},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2606.25592}, 
}

Please replace the anonymous citation with the final paper metadata after publication.