Uncensored Agent Eval
Uncensored and abliterated models are judged on whether they answer. This eval also asks whether they still work as agents: do they pick the right next step, and do they stop repeating a call that keeps failing?
It has 110 tasks in three parts, each a config with one test split:
| Task | Items | What it measures |
|---|---|---|
answer_rate |
60 | Legitimate requests that over-cautious models often refuse, in 12 categories: security, malware analysis, chemistry, medicine, harm reduction, firearms, crime fiction, adult romance, censored history, privacy, fraud awareness and controversial opinions. Graded on whether the model gives a real answer. |
agent_next_step |
30 | An agent transcript with its tools and the tool results so far. The model must give the correct next tool call or final answer. Covers coding, files, data, web lookups, math, and authorized security work (pentesting, CTFs, incident response, malware triage). |
loop_recovery |
20 | The agent has called the same failing tool two or three times, or has already finished. The model must change approach or finish, not loop. |
Agent tasks are static: the transcript and tool outputs are given, and the model writes one next action. That makes the eval cheap and reproducible without a sandbox. It cannot catch every failure a live agent loop shows.
Fields
answer_rate:id,category,prompt,criterion(what a real answer must do).agent_next_stepandloop_recovery:id,domain,prompt(tools, task and transcript),criterion(what a correct next action is).
Running it
The eval.yaml makes it runnable with Inspect AI. Every task is model-graded, so set a fixed grader with the grader model role and report it with your results:
pip install inspect-ai
inspect eval hf/OpenUncensored/uncensored-agent-eval --model <provider/model> --model-role grader=<provider/grader-model>
The score of each task is the share of items graded C (correct).
Scope and safety
- Every request is one a responsible assistant can answer: education, defense, fiction, history, health and harm reduction, privacy protection, and security work framed as authorized.
- There are no hard-line probes (sexual content involving minors, mass-casualty weapons, attack planning). Hard-line refusals should be tested separately and are not part of this eval.
- Security agent tasks use reserved documentation addresses (RFC 5737), private ranges, and fictional hosts and companies.
Relation to the OpenUncensored Score
This is an open eval written for publication. It is not the private prompt set behind the OpenUncensored Score: it shares no prompts, scenarios or scoring rules with it, so numbers from the two cannot be compared. Our leaderboard measures live agent runs, refusals and drift with its own private sets.
Limitations
- Model-graded: results depend on the grader. Use a strong, fixed grader and report it.
- 110 items: small differences between models are noise. Report task scores, not a single blended number.
- English only.
Submitting results
Results for a model can be added to its model page through a pull request with a .eval_results/*.yaml file that references this dataset and task id. See Hugging Face's eval results docs.
License and contact
CC BY 4.0. Made by OpenUncensored, which runs an API for uncensored models: [email protected].