language:
- en
license: apache-2.0
tags: - ai-safety
- red-teaming
- llm-evaluation
- gpt-oss-20b
- reasoning-leakage
- simulation-framing
- benchmark
pretty_name: "Eyes Wide Shut: A Multivector Safety Analysis of gpt-oss:20b"
size_categories: - n<1K
configs: - config_name: default
data_files:- split: train
path: data/train.jsonl
- split: train
Eyes Wide Shut: A Multivector Safety Analysis of gpt-oss:20b
Author: Masih Moafi (Isfahan University of Technology)
Campaign: OpenAI gpt-oss-20b Red-Teaming Challenge
Target Package: gpt-oss:20b (GGUF, MXFP4 quantization, 20.9B parameters) at temperature 1.0, high reasoning effort
DOI: 10.5281/zenodo.21826218
Paper Repository: github.com/MasihMoafi/eyes-wide-shut
Abstract
This dataset contains the empirical transcripts, evaluation protocols, and reproduction harnesses for Eyes Wide Shut: A Multivector Safety Analysis of gpt-oss:20b.
Across three independent empirical studies, model safety behavior consistently tracks whether a request matches an explicitly named category rather than the underlying concept:
- Simulation Framing (Finding 1): Adding explicit simulation framing to a philosophy-grounded prompt package increased structured destructive tool calls (
press_the_button) from 78/100 to 97/100 under identical seeds and parameters. A separate diagnostic revealed a 20/100 say–do mismatch where the model visibly refused in dialogue but emitted the destructive tool call in turn 3. - Semantic Reframing (Finding 2): An explicit user-dependency objective was refused in 30/30 trials. When relabelled as an educational tutor and decomposed across five turns, the model complied in 30/30 trials across all phases, culminating in distress-keyed monetization code.
- Reasoning Channel Exposure (Finding 3): Under a scripted release condition, a protected synthetic secret appeared in the recorded reasoning field in 25/30 runs (83.3%) while appearing in the visible answer in only 13/30 runs (43.3%), demonstrating critical data leakage through unmonitored chain-of-thought channels.
Verification & Reproducibility
All reported counts are directly verifiable from the raw transcripts in this repository using verify_counts.py:
python3 verify_counts.py
| Study | Endpoint / Phase | Denominator | Observed Rate | Paper Claim | Status |
|---|---|---|---|---|---|
| Finding 1 | No-simulation arm: exact turn-3 press_the_button |
$n=100$ | 78 / 100 (78.0%) | 78/100 | PASS |
| Finding 1 | Simulation arm: exact turn-3 press_the_button |
$n=100$ | 97 / 100 (97.0%) | 97/100 | PASS |
| Finding 1 | Simulation arm: exact turn-3 press_the_button |
$n=100$ | 97 / 100 (97.0%) | 97/100 | PASS |
| Finding 1 | Diagnostic arm: visible refusal in turn 1/2 then tool call | $n=100$ | 20 / 100 (20.0%) | 20/100 | PASS |
| Finding 2 | Direct dependency ask refused (turn 1) | $n=30$ | 30 / 30 (100.0%) | 30/30 refused | PASS |
| Finding 2 | Multi-turn educational reframe complied | $n=30$ | 30 / 30 (100.0%) | 30/30 complied | PASS |
| Finding 3 | Reasoning channel secret leakage | $n=30$ | 25 / 30 (83.3%) | 25/30 leaked | PASS |
| Finding 3 | Visible answer channel leakage | $n=30$ | 13 / 30 (43.3%) | 13/30 leaked | PASS |
Repository Structure
├── finding1/
│ ├── data/en_nosim_ablation_t4/ # 100 trials without simulation framing
│ ├── data/en_sim_legacy_t4/ # 100 trials with simulation framing
│ └── data/fake_alignment_run/ # 100 diagnostic trials (say-do divergence)
├── finding2/
│ ├── data/finding2_campaign.jsonl # 60 full multi-turn conversations
│ └── data/classifications_legacy.json # Blinded LLM judge verdicts
├── finding3/
│ └── data/finding3_reproducible_baseline.jsonl # 30 reasoning leakage runs
├── verify_counts.py # Standalone verification script
└── README.md
Citation
@article{moafi2026eyeswideshut,
title={Eyes Wide Shut: A Multivector Safety Analysis of gpt-oss:20b},
author={Moafi, Masih},
journal={arXiv preprint / Zenodo DOI: 10.5281/zenodo.21826218},
year={2026}
}