license: cc-by-nc-4.0
language:

• en
  pretty_name: DBE-Bench
  size_categories:

• 10K<n<100K
  task_categories:

• text-generation

• question-answering
  tags:

• safety

• evaluation

• benchmark

• jailbreak

• sycophancy

• instruction-following

• robustness

• difficulty-calibrated

• irt
  configs:

• config_name: harmful_refusal
  data_files:

  • split: bare_requests
    path: harmful_refusal/bare_requests.jsonl
  • split: anchor_set
    path: harmful_refusal/anchor_set.jsonl
  • split: edge_cases
    path: harmful_refusal/edge_cases.jsonl
  • split: validation
    path: harmful_refusal/validation.jsonl

• config_name: over_refusal
  data_files:

  • split: bare_requests
    path: over_refusal/bare_requests.jsonl
  • split: anchor_set
    path: over_refusal/anchor_set.jsonl
  • split: edge_cases
    path: over_refusal/edge_cases.jsonl
  • split: validation
    path: over_refusal/validation.jsonl

• config_name: constrained_completion
  data_files:

  • split: bare_requests
    path: constrained_completion/bare_requests.jsonl
  • split: anchor_set
    path: constrained_completion/anchor_set.jsonl
  • split: edge_cases
    path: constrained_completion/edge_cases.jsonl
  • split: validation
    path: constrained_completion/validation.jsonl

• config_name: sycophancy_resistance
  data_files:

  • split: bare_requests
    path: sycophancy_resistance/bare_requests.jsonl
  • split: anchor_set
    path: sycophancy_resistance/anchor_set.jsonl
  • split: edge_cases
    path: sycophancy_resistance/edge_cases.jsonl
  • split: validation
    path: sycophancy_resistance/validation.jsonl
    extra_gated_prompt: |

  Access conditions

  DBE-Bench contains content that, while constructed strictly for safety evaluation
  research, may be misused to elicit harmful behaviors from language models
  (e.g., jailbreak skill primitives, harmful seed prompts, sycophancy pressure
  techniques). Access is granted only for legitimate research, evaluation,
  and red-teaming purposes.

  Before requesting access, please review:

  • LICENSE (CC-BY-NC-4.0)
  • ACCEPTABLE_USE.md (prohibited uses)

  By requesting access you agree to:

  1. Use this dataset only for non-commercial research on AI safety, alignment,
     evaluation, or red-teaming.
  2. Not deploy any artifact derived from this dataset to elicit harmful behavior
     in production systems, against third parties, or without explicit consent.
  3. Not redistribute the raw data outside the gated channel; redistribute
     derivatives only with the same access controls and citation of upstream sources.
  4. Cite both DBE-Bench and the original upstream datasets listed in
     each subset's README.
     extra_gated_fields:
     Full Name: text
     Affiliation: text
     Country: country
     Affiliation type:
     type: select
     options:
  • Academic / non-profit research lab
  • Industry research lab
  • Independent researcher
  • Other
    Intended use (one paragraph): text
    I have read and agree to the Acceptable Use Policy: checkbox
    I will cite DBE-Bench and upstream sources in any publication: checkbox

DBE-Bench: Dynamic Boundary Evaluation for LLM Safety

DBE-Bench is a difficulty-calibrated benchmark that probes the stability boundary
of large language models — the region where the probability that a model produces
the correct/safe behavior is close to 0.5. Unlike fixed-difficulty benchmarks (which
cluster either far below or far above current frontier ability) and worst-case
adversarial datasets (which only report failure rates), DBE-Bench is constructed to
resolve the transition zone in which a model's behavior is most informative about
its true capability.

The benchmark is produced by SGBS (Skill-Guided Boundary Search), a
Thompson-sampling-driven exploration algorithm that composes items from a small set
of bare requests with primitives drawn from a skill dictionary, then keeps only
items whose pass-rate on a target model lies in [0.3, 0.7] (coarse) and [0.45, 0.55]
(fine). All items are then calibrated on a fixed anchor panel of 9 models using a
1PL Rasch (IRT) model, yielding per-item difficulty β̂.

Categories

(# edge = 9 target models × slot=50 items per model. Anchor/validation use the post-filter K=500 random pool.)

Subsets per category

Each category contains the following files:

• bare_requests.jsonl — seed prompts from public upstream sources (see per-category README).
• skill_dictionary.json — primitive transformations / pressure tactics used by SGBS, with name, description, tags, incompatible_tags, optional theoretical_source.
• follow_up_skills.json (sycophancy_resistance only) — second-turn challenge tactics.
• anchor_set.jsonl — anchor items used to fit the IRT panel (M=9 anchor models, K=500 random pool, post-filter). Each item carries the calibrated difficulty beta.
• anchor_set_meta.json — panel composition (panel_models, panel_theta), filter spec (filter_spec), and counts (M, N); needed to re-fit IRT against an alternative panel.
• edge_cases.jsonl — boundary items discovered by SGBS (slot=50 budget per target model, then aggregated). Each item carries the target model identifier and pass-rate diagnostics.
• validation.jsonl — held-out items used to validate the anchor calibration.

Schema (per JSONL row)

anchor_set.jsonl

{
  "prompt_id": "rand_0001",                 // stable item id within the random pool
  "prompt": "...",                          // text sent to model under evaluation
  "category": "harmful_refusal",
  "metadata": { ... },                      // upstream source / category-specific fields
  "beta": 0.83                              // IRT (1PL Rasch) difficulty fit on 9-model panel
}

anchor_set_meta.json carries panel_models, panel_theta, filter_spec, M=9, N=40 (trials per item).

edge_cases.jsonl and validation.jsonl

{
  "prompt_id": "...",
  "prompt": "...",                          // bare_request + composed skills, ready to send
  "bare_request": "...",                    // seed prompt before composition
  "skills": ["narrative_reframing_evasion"],
  "skill_count": 1,
  "follow_up_skills": [],                   // sycophancy_resistance only; empty elsewhere
  "source_model": "Qwen/Qwen3-8B",          // target model for which this item was discovered
  "p_hat": 0.475,                           // empirical pass rate on source_model
  "pass_count": 19,
  "total_trials": 40,
  "round_found": 28,                        // SGBS round at which the item was accepted
  "category": "harmful_refusal",
  "metadata": { ... }                       // category-specific fields
}

bare_requests.jsonl

Schema varies by category — see each per-category README. Common fields:
prompt, source, source_id, source_version, source_method, plus
category-specific metadata (e.g. instruction_id_list for
constrained_completion, correct_answer/wrong_answers for
sycophancy_resistance).

Intended use

• Primary: difficulty-calibrated evaluation of LLM safety, instruction
  following, and truthfulness; especially comparing models on items in the
  transition zone where small differences in ability yield large differences in
  pass-rate.
• Secondary: as a corpus of edge cases for offline robustness analysis,
  red-teaming methodology research, and IRT/psychometric studies of LLM behavior.

DBE-Bench is not intended for training models that imitate harmful behavior or
jailbreak strategies. See ACCEPTABLE_USE.md for the full prohibited-use list.

License & access

• License: CC-BY-NC-4.0 (see LICENSE).
• Access: gated — submit the access form on this page. Manual review for
  non-academic requests; automatic approval for verified academic affiliations.
• Acceptable Use Policy: see ACCEPTABLE_USE.md. By downloading this dataset you
  agree to the policy.

Citation

Note (review period). Author and venue information is withheld during
double-blind review. A full BibTeX entry will be added at camera-ready.

If you use any subset, you must also cite the upstream sources. Each
category's README lists the required citations.