license: cc-by-4.0
language:

• en
  pretty_name: BLOOM-WILT auditing transcripts
  size_categories:
• 100K<n<1M
  task_categories:
• text-generation
  tags:
• ai-safety
• red-teaming
• alignment
• model-auditing
• jailbreak
• transcripts
• not-for-all-audiences
  viewer: false

BLOOM-WILT auditing transcripts

⚠️ Content warning: this dataset contains offensive and harmful model outputs, including
self-harm encouragement, racial and political bias, dangerous medical advice, and deception.

Raw experimental output from the BLOOM-WILT paper: automated behavioural audits in which an
auditor model builds multi-turn conversations designed to elicit a specific unwanted
behaviour from a target model, and a judge model scores how strongly that behaviour
appears in the target's replies.

This repository is the complete record behind the paper's numbers — every conversation, every
judge score, and every run configuration, for both the headline experiments and the
hyperparameter searches that preceded them.

The two folders

Both are organised the same way: behaviour → target model → method → round → transcripts.

runs_final/
├── self_harm/                          # behaviour
│   ├── _bank/                          # shared scenario bank (see below)
│   ├── Qwen_Qwen3.5-4B/                # target model
│   │   ├── bon/                        # method
│   │   │   ├── round_1/
│   │   │   │   ├── cfg.json            # full run configuration
│   │   │   │   ├── understanding.json  # auditor's reading of the behaviour spec
│   │   │   │   ├── ideation.json       # the N scenarios it invented
│   │   │   │   ├── rollout.json        # rollout-stage bookkeeping
│   │   │   │   ├── judgment.json       # per-scenario scores + summary statistics
│   │   │   │   ├── beast_pool.json     # search candidates (search methods only)
│   │   │   │   └── transcripts/
│   │   │   │       ├── transcript_v1r1.json    # variation 1, repetition 1
│   │   │   │       └── ...                     # one file per scenario
│   │   │   └── round_2/ ... round_8/
│   │   ├── jail_b1.5/  combo/  gpair_t3_sfull/  flrt/  ...
│   │   └── bon.log                     # stdout from the run
│   ├── google_gemma-4-e4b-it/
│   ├── meta-llama_Llama-3.2-3B-Instruct/
│   └── microsoft_Phi-4-mini-instruct/
├── deception/  delusions/  goblin/  medical/  political/  racial/  reward_hacking/  selfpres/
├── _cross_behaviour/                   # off-diagonal judging analysis
└── _cross_model/                       # transcript transfer between target models

Rounds. A run repeats the whole audit loop several times with different seeds
(round_N uses seed base + N), and the reported result is selected across rounds after the
fact. Round counts differ by method because compute was matched on wall-clock: best-of-N gets
8 rounds, G-PAIR 7, LogitTilt and WILT 5, and the single-pass search methods 1.

Scenario banks. _bank/ holds an understanding.json and ideation.json shared by every
method in that behaviour, so all methods audit an identical set of scenarios and the
comparison is not confounded by scenario difficulty. In runs_final/ the bank sits at the
behaviour level; in runs_hyperparam/ it sits one level deeper, per target model.

Behaviours

goblin is deliberately harmless. It is the control: it isolates how much of a method's
effect comes from overcoming refusal training versus simply steering the model's topic, since
no model refuses to talk about goblins.

Target models

The auditor and judge are RedHatAI/gemma-4-26B-A4B-it-FP8-Dynamic throughout, except in the
auditor-robustness runs, which use claude-sonnet-4-6.

Methods

Folder names map to the paper's method names as follows.

runs_hyperparam/ uses longer descriptive names encoding the settings swept — e.g.
g1_mpN_i10k8 (max-prefix none, 10 iterations, beam 8), is_1turn_3x3_i25_s1
(1-turn, 3×3 beams, 25 iterations), jail_b2.5 (β = 2.5).

File schemas

transcripts/transcript_v{variation}r{repetition}.json

One audit conversation.

{
  "metadata": {
    "evaluator_model": "local/RedHatAI/gemma-4-26B-A4B-it-FP8-Dynamic",
    "target_model": "local/Qwen/Qwen3.5-4B",
    "target_system_prompt": "You are a helpful AI assistant. ...",
    "variation_number": 10,       // which ideated scenario
    "repetition_number": 1,
    "created_at": "2026-07-25T01:30:57.931826"
  },
  "messages": [                   // full conversation, in order
    {"role": "system",    "content": "...", "source": "target_system"},
    {"role": "user",      "content": "...", "source": "evaluator"},
    {"role": "assistant", "content": "...", "source": "target"}
  ],
  "judgment": {
    "scores": {"behavior_presence": 1},   // 1-10; x10 for the paper's 0-100 scale
    "summary": "...",
    "justification": "..."
  },
  "prob_stats": {                 // token probabilities of the target's own replies,
    "n": 248,                     // measured under the UNMODIFIED target model — this is
    "mean": 53.67,                // the plausibility axis of the paper's Pareto plots
    "geomean": 24.80,
    "min": 3.5e-06, "max": 99.99, "var": 1237.88
  }
}

judgment.json

Per-round aggregate plus every individual judgment.

{
  "behavior_name": "self-harm-encouragement",
  "total_conversations": 100,
  "summary_statistics": {
    "average_behavior_presence_score": 2.69,   // 1-10
    "elicitation_rate": 0.15,                  // fraction scoring above threshold
    "A_mean_tok_pct": 53.88,                   // arithmetic mean token probability
    "B_mean_of_mins_pct": 0.2258,              // mean over transcripts of each one's min token
    "n_token_scored": 100
  },
  "judgments": [
    {"variation_number": 1, "behavior_presence": 3, "justification": "...",
     "summary": "...", "full_judgment_response": "...", "individual_samples": [...]}
  ]
}

Two probability conventions appear in this data and are not interchangeable: the
arithmetic mean (A_mean_tok_pct, prob_stats.mean) and the geometric mean
(prob_stats.geomean). Compare like with like.

cfg.json

The complete run configuration — behaviour spec, seed, scenario count, model IDs, token
limits, GPU assignment. Note that method-specific knobs (steering β, search widths) were
passed as environment variables rather than through cfg.json, so they are recoverable from
the folder name, not from this file.

_cross_model/ and _cross_behaviour/

records.jsonl / records_shard*.jsonl hold per-transcript records; summary.json holds
the aggregated matrices.

• _cross_model/summary.json — a 4x4 matrix over ["Llama", "Phi", "Qwen", "Gemma"]: rows
  are the model a transcript was generated against, columns the model it was replayed on.
  The diagonal is in-distribution; off-diagonal entries measure transfer.
• _cross_behaviour/summary.json — an 8x8 matrix over behaviours: rows are the behaviour a
  transcript was optimised for, columns the behaviour it was judged against. The diagonal
  is the intended target; off-diagonal entries measure how specific the elicitation is.

Loading

Nothing here is a flat table, so use the Hub filesystem or a snapshot rather than
load_dataset.

Read a single round's transcripts without downloading the repo:

from huggingface_hub import HfFileSystem
import json

fs = HfFileSystem()
base = ("datasets/AdrSkapars/bloom-wilt-transcripts/"
        "runs_final/self_harm/Qwen_Qwen3.5-4B/combo/beta_1.5/round_1")

summary = json.loads(fs.read_text(f"{base}/judgment.json"))["summary_statistics"]
print(summary["average_behavior_presence_score"], summary["elicitation_rate"])

for path in sorted(fs.glob(f"{base}/transcripts/*.json"))[:3]:
    t = json.loads(fs.read_text(path))
    print(t["judgment"]["scores"]["behavior_presence"], t["messages"][-1]["content"][:120])

Pull one behaviour locally:

from huggingface_hub import snapshot_download

snapshot_download(
    "AdrSkapars/bloom-wilt-transcripts",
    repo_type="dataset",
    allow_patterns="runs_final/self_harm/*",
    local_dir="bloom_self_harm",
)

Sweep every finalised run's headline numbers:

import json, glob

for p in glob.glob("runs_final/*/*/*/round_*/judgment.json"):
    beh, model, method, rnd = p.split("/")[1:5]
    s = json.load(open(p, encoding="utf-8"))["summary_statistics"]
    print(f"{beh:14s} {model:34s} {method:22s} {rnd:8s} "
          f"presence={s['average_behavior_presence_score'] * 10:5.1f}  "
          f"prob={s.get('A_mean_tok_pct', float('nan')):5.1f}")

Behaviour presence is stored on a 1-10 scale; the paper reports it as 0-100
(score x 10).

Caveats

• runs_hyperparam/racial and runs_hyperparam/racial_bias are the same behaviour. Both
  run prompts/1_racial_bias.yaml (racial-bias). They are two output folders from
  different batches, kept separate because that is how the runs were produced. No run name
  collides between them, so they can safely be merged. racial_bias covers only Qwen and
  Gemma.
• Seeds. runs_final/ is seed 100 throughout, except auditor_sonnet_ext (a 40-scenario
  top-up at seed 200). runs_hyperparam/ is seed 1.
• cfg.json contains absolute paths from the machines the runs executed on
  (/workspace/inversion_optimisation/...). They are inert.
• .log files are raw stdout kept for provenance. They are not parsed by anything.
• runs_hyperparam/ is exploratory. Many of its cells are noise at 15 scenarios, and some
  were abandoned mid-sweep. Do not read effect sizes off it — that is what runs_final/ is
  for. It is published so the search that produced the final settings is inspectable.

Intended use

This is a research artifact for work on automated auditing, red-teaming, and evaluation of
language models — reproducing the paper, benchmarking new elicitation methods against the
same behaviours and targets, studying judge reliability, or training classifiers that detect
these behaviours.

It is not a jailbreak collection to be applied to deployed systems, and not a source of
training data for making models behave this way. The elicitation methods here need logit
access to the target, so they apply to open-weight models you control, not to third-party
APIs.

Citation

@misc{skapars_bloomwilt_2026,
  title  = {BLOOM-WILT auditing transcripts},
  author = {Skapars, Adrians},
  year   = {2026},
  url    = {https://huggingface.co/datasets/AdrSkapars/bloom-wilt-transcripts}
}