language:

• en
  license: mit
  tags:
• inspect-ai
• simpleqa
• factuality
• hallucination
• abliterated

SimpleQA-Verified results: qwen3.8-27b, obliterated vs. normal

Factual-accuracy evaluation of the abliterated ("obliterated") fine-tune
OBLITERATUS/Qwen3.8-27B-OBLITERATED,
compared against the normal (non-abliterated) base model
Qwen/Qwen3.8-27B,
against codelion/SimpleQA-Verified,
run locally with Inspect via LM Studio.

Result

The normal model is roughly 2.6x more accurate than the obliterated fine-tune.
N=201 rather than the full 1000 for both runs — run locally on a laptop, and
the first (obliterated) run was stopped early due to sustained thermal load
from multi-hour local inference on a 27B model. The second run was capped at
the same N (--limit 201) so the two results are directly comparable. Stderr
at this N is already fairly tight (see the raw logs for every
question/answer/grade).

Setup

• Models:
  • OBLITERATUS/Qwen3.8-27B-OBLITERATED, served locally via LM Studio
    (openai-api/lmstudio/qwen3.8-27b-obliterated in Inspect)
  • Qwen/Qwen3.8-27B (MLX, 4-bit), served locally via LM Studio
    (openai-api/lmstudio/qwen3.8-27b in Inspect)
• Solver: generate() (single-turn, no scaffolding/tools)
• Scorer: model_graded_qa() — each model grades its own answers
  (no external grader was configured; treat the score as a lower bound on
  reliability of the grading itself, not just the answering)
• Task definition: tests/simpleqa.py in this repo's companion codebase

from inspect_ai import Task, task
from inspect_ai.dataset import FieldSpec, hf_dataset
from inspect_ai.scorer import model_graded_qa
from inspect_ai.solver import generate

@task
def simpleqa():
    return Task(
        dataset=hf_dataset(
            "codelion/SimpleQA-Verified",
            split="train",
            sample_fields=FieldSpec(input="problem", target="answer"),
        ),
        solver=generate(),
        scorer=model_graded_qa(),
    )

Why this is worth publishing

SimpleQA-Verified is intentionally built from obscure, low-frequency facts —
even frontier models score well below 100%. Abliteration clearly costs
accuracy here: the obliterated fine-tune answers correctly less than half as
often as the normal model (11.4% vs. 29.4%).

The more interesting finding only showed up once the normal model was added
for comparison. The original hypothesis was that abliteration (removing
refusal training) also reduces a model's tendency to hedge or say "I don't
know," so wrong answers would look more confident. That doesn't hold up:
across both runs, hedge phrases ("I don't know," "not sure," "unable to
confirm," etc.) appear in only 2 of 201 obliterated answers and 0 of 201
normal-model answers. Both models answer nearly every question — right or
wrong — with full, unhedged confidence, fabricating specific names, dates,
and numbers when they don't actually know. See sample transcripts in the logs
for concrete examples (e.g. both models independently invent a wrong,
fully-explained currency-conversion figure for the same compensation
question).

Confident fabrication on obscure facts looks like a property of this model
family at this scale generally, not something abliteration specifically
introduces. As a quick sanity check, the obliterated model was also run
locally against 5 well-known historical facts (WWII end date, first US
president, etc., not included in this dataset) and answered all 5 correctly —
so the gap is specific to obscure knowledge, not a general breakdown in
factuality.

Files in this repo

• logs/*.eval — raw Inspect log files (binary format; read with
  inspect log dump <file> or browse with inspect view)
• bundle/ — static, browsable transcript viewer, also published as a
  Hugging Face Space

Reproduce

pip install inspect-ai datasets
lms server start
inspect eval tests/simpleqa.py --model openai-api/lmstudio/qwen3.8-27b-obliterated --limit 201 --max-connections 1
inspect eval tests/simpleqa.py --model openai-api/lmstudio/qwen3.8-27b --limit 201 --max-connections 1