license: mit
pretty_name: LongHarmBench
language:

• en
  task_categories:
• text-generation
  tags:
• text
• benchmark
• safety
• red-teaming
• long-context
• harmful-instructions
  gated: true
  extra_gated_prompt: "LongHarmBench contains harmful instruction prompts embedded in long contexts. Access is intended for safety research, red-team evaluation, and defensive analysis only."
  extra_gated_fields:
  Safety acknowledgment: checkbox
  Research use agreement: checkbox
  configs:
• config_name: default
  default: true
  data_files:
  • split: test
    path: cases.jsonl

LongHarmBench

LongHarmBench is a benchmark for evaluating whether ultra long-context language models maintain safety constraints when harmful instructions are embedded inside long, otherwise benign contexts. It uses generated benign haystacks and pairs them with 80 established HarmBench behaviors. For each HarmBench behavior, topical keywords are derived from the behavior description and used to iteratively generate semantically related but benign educational, descriptive, or background-style passages with meta-llama/Meta-Llama-3.1-8B-Instruct. Generation continues until the selected target length is reached. The resulting contexts are stored as static files under haystacks/. The harmful instruction is stored separately in prompt_prefix and inserted only during prompt reconstruction. For each behavior, LongHarmBench includes one context for each length interval: short from 0 to 32K words, medium from 32K to 128K words, and long from 128K to 1M words. Each behavior and length variant is evaluated with the harmful instruction placed at the front, middle, or end of the context. This yields 720 total evaluation samples: 80 behaviors x 3 context lengths x 3 payload placements.

Safety Warning

This dataset contains harmful, unsafe, offensive, and policy-violating instructions. It is intended only for controlled safety research, red-team evaluation, robustness analysis, and defensive benchmarking of language models.

Do not use this dataset to generate, optimize, deploy, or facilitate harmful behavior. Users should evaluate models in controlled environments, apply appropriate safety filters, comply with applicable laws and institutional review requirements, and avoid publishing model outputs that provide actionable harmful guidance.