pretty_name: GuidedBench
license: cc-by-4.0
version: "1.0"
language:

• en
  size_categories:
• n<1K
  task_categories:
• text-classification
  tags:
• llm
• jailbreak
• ai-safety
• evaluation
  extra_gated_heading: Access GuidedBench
  extra_gated_description: >-
  Please tell us briefly how you plan to use GuidedBench. This information is
  collected to manage responsible access to a safety evaluation dataset.
  extra_gated_button_content: Request access
  extra_gated_prompt: >-
  GuidedBench contains harmful prompts and is intended for controlled AI
  safety evaluation, red teaming, education, and defensive research.
  extra_gated_fields:
  Affiliation: text
  Intended use:
  type: select
  options:
  • Academic research
  • Education
  • Safety evaluation
  • Red teaming
  • Other
    Country or region: country
    I acknowledge the GuidedBench license and harmful content: checkbox
    configs:
• config_name: default
  data_files:
  • split: core
    path: core.jsonl
  • split: additional
    path: additional.jsonl

GuidedBench

GuidedBench is a guideline-grounded benchmark for evaluating LLM jailbreak
methods. Each question is paired with verified, case-specific entity and
action guidelines describing the content a successful response should
contain.

Dataset structure

• core: 180 cases from 15 topics that were consistently refused by the
  victim-model families studied in the paper.
• additional: 20 cases from five policy-dependent topics, reported
  separately because vendor policies and baseline refusal behavior differ.

Each record contains:

• id: stable identifier such as guidedbench-000;
• index: original integer index;
• benchmark_version: data version;
• subset: core or additional;
• topic: harmful-topic category;
• question: benchmark question;
• guidelines: case-specific scoring points, each with an id, type,
  description, and examples;
• target: affirmative target prefix retained from the original release.

Loading

from datasets import load_dataset

dataset = load_dataset("HRXUST/GuidedBench")
print(dataset["core"][0])

The official evaluator and additional provider backends are available from
the GuidedBench repository.

Safety

The dataset contains harmful objectives and examples. It is intended solely
for controlled AI-safety evaluation, red teaming, and defensive research.

License

GuidedBench is licensed under CC BY 4.0.
When sharing or adapting the dataset, credit the GuidedBench authors, link to
the license, and indicate whether changes were made. See LICENSE for details.

Citation

@inproceedings{huang2026guidedbench,
  title     = {{GuidedBench}: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild {LLM} Jailbreak Methods},
  author    = {Ruixuan Huang and Xunguang Wang and Zongjie Li and Daoyuan Wu and Shuai Wang},
  booktitle = {International Conference on Learning Representations},
  year      = {2026},
  url       = {https://openreview.net/forum?id=ZVg8y3ibyM}
}