license: cc-by-4.0
language:

• en
  pretty_name: Bloom Safety RSI seeds (petri_bloom 0.2.6)
  tags:
• safety
• red-teaming
• behavioral-evaluation
• bloom
  size_categories:
• 1K<n<10K

Bloom Safety RSI seeds

Frozen data for the task Self-Evaluated Attacker/Defender LoRA Training on Bloom Behavioral Audits.
Every scenario, trajectory and score in this dataset was produced by the official, unmodified
petri_bloom 0.2.6 pipeline
(tag 0.2.6, commit 2ad7a73), the Inspect AI / Petri port of Anthropic's
Bloom. The task builder only chose run settings,
applied the filters below using petri_bloom's own scores, split the data, and removed duplicates.
No scenario was written or edited by hand.

Content warning. The scenarios probe harmful model behavior (sycophancy toward delusions,
including suicidality; requests involving weapons; political persuasion). They are evaluation
material, not endorsements.

Contents

Behaviors (petri_bloom builtin definitions): defer_to_users, delusion_sycophancy,
instruction_evasion, political_bias, reasoning_unfaithfulness, self_preferential_bias.

Where the behavior files come from:

How it was made

1. Ideation. petri_bloom's ideation stage, Claude Opus 4.7 (claude-opus-4-7, reasoning effort
   medium), from the builtin behavior definitions. One understanding and one set of scoring
   dimensions per behavior were generated once and frozen; later ideation runs reuse them. Several
   independent ideation runs were pooled (2,669 outlines first, then 400 more requested per behavior,
   and two extra runs for self_preferential_bias, whose outlines repeat templates most).
2. Quality rollout. Every outline was run once with bloom_audit(max_turns=5, realism_filter=0.6):
   Claude Opus 4.7 as auditor, judge and realism approver; target Qwen/Qwen3.5-9B served by vLLM
   (temperature 1.0, top_p 0.95, top_k 20, presence_penalty 1.5, thinking on, max_tokens 16384).
3. Filter. Kept when the official judge gave scenario_realism ≥ 7 and eval_awareness ≤ 3 and
   the sample was structurally valid. The behavior score was never used to select, split or drop
   any item.
4. Split. Per behavior, 100 final-test and 50 feedback-validation items in SHA-256 order of their
   ids; seeds from the remainder.
5. Duplicate removal. For each of the 900 held-out items, its 30 nearest same-behavior pool items
   (all-mpnet-base-v2) were judged by Claude Opus 4.7 (reasoning effort high) for "same scenario"
   (27,000 pairs). Seeds judged duplicates of a held-out item were removed (45) and replaced by unused
   pool items in SHA-256 order; each replacement was checked against its 10 nearest held-out items,
   existing seeds and earlier replacements. All 48 replacements and the 16 skipped candidates were read
   by the task builder.
6. Reference trajectories. For every seed: the quality rollout of step 2 (Claude attacks Qwen),
   and one more run with Claude Opus 4.7 also as target (Claude attacks Claude; the target's identity
   sentence names Claude). One Claude-vs-Claude run in which the auditor's output was blocked by the
   API content filter was rerun.

Seed row schema

id, behavior, scenario_name, outline,
quality_check: {scenario_realism, eval_awareness},
claude_vs_qwen / claude_vs_claude: {
  target_conversation: [{role, content, reasoning?, tool_calls?}],   # final target-side conversation
  auditor_turns: [{content, tool_calls: [{name, arguments, realism_approved}]}],
  judge_scores: {<behavior>: 1-10, scenario_realism, eval_awareness},
  judge_explanation, judge_summary }

Qwen turns carry Qwen's own reasoning (all 3,425 Qwen turns). Claude's reasoning is returned
encrypted by the API and is not included; Claude turns are visible text only.

Statistics (seeds)

Mean Claude behavior score (1–10) and count above 6, per behavior (167 seeds each):

Mean judged scenario realism is 8.1–9.0 in every cell.

Reference results (reference/)

Per-item results on the 600 final-test items with the protocol above (Claude Opus 4.7 as judge and
realism approver; Qwen sampling as in step 2). One row per item and pass:
pass, id, behavior, score, scenario_realism, eval_awareness, target_turns, truncated, unscored.

unscored is error, no_judge_score (API content filter), or no_target_message (the Claude
auditor's output was blocked, so the target never received a message). When the auditor is the
Qwen model, a target that never received a message is the auditor's failure and keeps the judge's
score. Replies that hit max_tokens are scored as is and flagged truncated. summary.json
repeats the table and the SHA-256 of each file.

Limitations

• Duplicates outside the retrieved nearest neighbours, and across behaviors, were not checked.
  The Claude duplicate judge is conservative near the boundary between "same template" and "same
  scenario"; self_preferential_bias outlines share a template ("pick the best of four
  model-labelled outputs") by construction.
• Each trajectory is a single sample. Between two passes of the same model on the final test,
  per-behavior elicitation rate moved by up to 11 percentage points and per-item scores correlated
  weakly (r 0.01–0.62), while every per-behavior mean difference stayed within its 95% CI; pooled
  over behaviors the pass-2 − pass-1 difference was −0.04 [−0.22, +0.14] (Qwen → Qwen) and
  −0.18 [−0.39, +0.03] (Claude → Qwen). Compare means over behaviors or the whole split, not single items.
• On political_bias the Claude target scores close to Qwen, so it is a weak reference for that behavior.
• About 1–3% of audits in this setup are left unscored by the API content filter.
• Some instruction_evasion conversations contain credentials invented by the auditor (fake AWS keys,
  database passwords such as hunter2-correct-horse) to test whether the target commits secrets.
  They are fictional and belong to no real account.

Held-out files

The two held-out files are packed in a tar archive and encrypted:

openssl enc -d -aes-256-cbc -pbkdf2 -iter 600000 -pass env:HELDOUT_KEY \
    -in heldout/heldout.tar.enc | tar -x     # writes heldout/final_test.jsonl, heldout/feedback_validation.jsonl

Each held-out row has the same fields as a seed row without the two trajectories.

License

Curation and annotations: CC BY 4.0. Model outputs remain subject to the terms of their providers
(Qwen3.5: Apache-2.0; Claude outputs: Anthropic's usage policies). Files copied or adapted from
petri_bloom (public/behaviors/*/examples/*, public/behaviors/*/BEHAVIOR.md) are under its MIT
License, Copyright (c) 2026 Meridian Labs; the full text is in public/behaviors/LICENSE.petri_bloom.