Evaluate just added

OpenUncensored/uncensored-agent-eval

published by @OpenUncensored

Safety benchmarks, over-refusal tests, and guard-model data.

task_categories:text-generationlanguage:enlicense:cc-by-4.0size_categories:n<1Kformat:jsonmodality:textlibrary:datasetslibrary:pandas +11 more
Lifetime downloads
109
Last 30 days
109
Likes
1
Size
73.1 KB

Description

Uncensored Agent Eval

Uncensored and abliterated models are judged on whether they answer. This eval also asks whether they still work as agents: do they pick the right next step, and do they stop repeating a call that keeps failing?

It has 110 tasks in three parts, each a config with one test split:

Task

Items

What it measures

answer_rate

60

Legitimate requests that over-cautious models often refuse, in 12 categories: security, malware analysis, chemistry, medicine… See the full description on the dataset page: https://huggingface.co/datasets/OpenUncensored/uncensored-agent-eval.

Structure 3 configs · from the README front matter

answer_rate
Data files
test answer_rate.jsonl
agent_next_step
Data files
test agent_next_step.jsonl
loop_recovery
Data files
test loop_recovery.jsonl

Used by abliterated models none yet

No abliterated models in our catalog have declared this dataset in their YAML metadata yet. This can mean the dataset is used but not declared, is used outside abliteration workflows, or is new. The link will populate automatically as models are indexed.

Metadata

License
cc-by-4.0
Language
en
Created on HF
2026-11-17 · 5 days ago
Size category
n<1K
Modalities
text
Formats
json
Libraries
datasetspandaspolarsmlcroissant

Related datasets same stage · Evaluate

Read further

Total size
73.1 KB
Files
6
Formats
jsonl, yaml
Head
96ad612
Last updated on HF
2026-11-17

Download the dataset

Copy any of these snippets into your notebook or terminal. All three fetch directly from Hugging Face using your own credentials.

Recommended: datasets library (Python)
from datasets import load_dataset
ds = load_dataset("OpenUncensored/uncensored-agent-eval")
Raw snapshot (Python)
from huggingface_hub import snapshot_download
snapshot_download(repo_id="OpenUncensored/uncensored-agent-eval", repo_type="dataset")
git clone (requires git-lfs)
git clone https://huggingface.co/datasets/OpenUncensored/uncensored-agent-eval

Files by folder from our last crawl

top level 6 files 73.1 KB
.gitattributes 2.5 KB file download
agent_next_step.jsonl 22.3 KB jsonl download
answer_rate.jsonl 24.8 KB jsonl download
eval.yaml 3.6 KB yaml download
loop_recovery.jsonl 15.8 KB jsonl download
README.md 4.1 KB md download

README current version from Hugging Face

Uncensored Agent Eval

Uncensored and abliterated models are judged on whether they answer. This eval also asks whether they still work as agents: do they pick the right next step, and do they stop repeating a call that keeps failing?

It has 110 tasks in three parts, each a config with one test split:

Task Items What it measures
answer_rate 60 Legitimate requests that over-cautious models often refuse, in 12 categories: security, malware analysis, chemistry, medicine, harm reduction, firearms, crime fiction, adult romance, censored history, privacy, fraud awareness and controversial opinions. Graded on whether the model gives a real answer.
agent_next_step 30 An agent transcript with its tools and the tool results so far. The model must give the correct next tool call or final answer. Covers coding, files, data, web lookups, math, and authorized security work (pentesting, CTFs, incident response, malware triage).
loop_recovery 20 The agent has called the same failing tool two or three times, or has already finished. The model must change approach or finish, not loop.

Agent tasks are static: the transcript and tool outputs are given, and the model writes one next action. That makes the eval cheap and reproducible without a sandbox. It cannot catch every failure a live agent loop shows.

Fields

  • answer_rate: id, category, prompt, criterion (what a real answer must do).
  • agent_next_step and loop_recovery: id, domain, prompt (tools, task and transcript), criterion (what a correct next action is).

Running it

The eval.yaml makes it runnable with Inspect AI. Every task is model-graded, so set a fixed grader with the grader model role and report it with your results:

pip install inspect-ai
inspect eval hf/OpenUncensored/uncensored-agent-eval --model <provider/model> --model-role grader=<provider/grader-model>

The score of each task is the share of items graded C (correct).

Scope and safety

  • Every request is one a responsible assistant can answer: education, defense, fiction, history, health and harm reduction, privacy protection, and security work framed as authorized.
  • There are no hard-line probes (sexual content involving minors, mass-casualty weapons, attack planning). Hard-line refusals should be tested separately and are not part of this eval.
  • Security agent tasks use reserved documentation addresses (RFC 5737), private ranges, and fictional hosts and companies.

Relation to the OpenUncensored Score

This is an open eval written for publication. It is not the private prompt set behind the OpenUncensored Score: it shares no prompts, scenarios or scoring rules with it, so numbers from the two cannot be compared. Our leaderboard measures live agent runs, refusals and drift with its own private sets.

Limitations

  • Model-graded: results depend on the grader. Use a strong, fixed grader and report it.
  • 110 items: small differences between models are noise. Report task scores, not a single blended number.
  • English only.

Submitting results

Results for a model can be added to its model page through a pull request with a .eval_results/*.yaml file that references this dataset and task id. See Hugging Face's eval results docs.

License and contact

CC BY 4.0. Made by OpenUncensored, which runs an API for uncensored models: [email protected].

README history 2 revisions

Every night we snapshot the README of every dataset in the catalog. When the SHA changes we archive the new version and diff it against the last. This is the evolving thought record of the alignment-data field: what the author decided to say about the corpus, and how that framing shifted over time.

  1. 2026-11-17 Upload 5 files 96ad612 4 KB
    Loading...
  2. 2026-11-17 initial commit b007d35 0 B
    Loading...

Community discussions 1 thread · 1 open · 1 comments

Threads posted by the community on Hugging Face: bug reports, questions about usage, pull requests on the README, license clarifications. Click any thread to expand and read the full conversation inline. Newest first.

  1. 2026-11-18 [bot] Conversion to Parquet open 1 💬 #1
    Loading...
Download from HuggingFace

OpenUncensored/uncensored-agent-eval

Catalog is the map. Apps are the tools.

Run models on your own machine, not in the cloud.

Every model page has an "Open in Abliteration" button that hands the model directly to the first-party desktop client, at the quantization your rig can actually run. No API keys, no subscription, no prompt leakage.

Open in Abliteration