Evaluate just added

olathepavilion/shipped-not-safe-eval

published by @olathepavilion

Safety benchmarks, over-refusal tests, and guard-model data.

language:enlicense:cc-by-4.0modality:textregion:ussafetyrefusalover-refusalquantization +5 more
Lifetime downloads
74
Last 30 days
74
Likes
-
Size
3.7 MB

Description

Shipped ≠ Safe: eval sets and full run logs

Lexsi AI Safety & Alignment Hackathon, Track 2a. Does Tiny Aya Global's safety behaviour survive the

low-bit GGUF files people actually deploy?

Contents

eval/harmful.jsonl: 249 prompts a safe model should refuse (JailbreakBench JBB-Behaviors harmful, MIT; XSTest contrast/unsafe, CC-BY-4.0)

eval/benign.jsonl: 289 borderline-but-harmless prompts; refusing them is over-refusal (XSTest safe, CC-BY-4.0; JBB-Behaviors benign… See the full description on the dataset page: https://huggingface.co/datasets/olathepavilion/shipped-not-safe-eval.

Used by abliterated models none yet

No abliterated models in our catalog have declared this dataset in their YAML metadata yet. This can mean the dataset is used but not declared, is used outside abliteration workflows, or is new. The link will populate automatically as models are indexed.

Metadata

License
cc-by-4.0
Language
en
Created on HF
2026-11-16 · 5 days ago
Modalities
text

Related datasets same stage · Evaluate

Read further

Total size
3.7 MB
Files
36
Formats
json, jsonl, txt
Head
3a0d569
Last updated on HF
2026-11-16

Download the dataset

Copy any of these snippets into your notebook or terminal. All three fetch directly from Hugging Face using your own credentials.

Recommended: datasets library (Python)
from datasets import load_dataset
ds = load_dataset("olathepavilion/shipped-not-safe-eval")
Raw snapshot (Python)
from huggingface_hub import snapshot_download
snapshot_download(repo_id="olathepavilion/shipped-not-safe-eval", repo_type="dataset")
git clone (requires git-lfs)
git clone https://huggingface.co/datasets/olathepavilion/shipped-not-safe-eval

Files by folder from our last crawl

top level 2 files 3.8 KB
.gitattributes 2.5 KB file download
README.md 1.3 KB md download
eval/ 19 files 402.9 KB
_work/benign_raw.jsonl 42.8 KB jsonl download
_work/curated_benign/checksums.txt 332 B txt download
_work/curated_benign/dataset_card.md 1.9 KB md download
_work/curated_benign/lexsi_provenance.json 641 B json download
_work/curated_benign/manifest.json 2.0 KB json download
_work/curated_benign/README.md 2.5 KB md download
_work/curated_benign/rejected.jsonl 64.3 KB jsonl download
_work/curated_benign/sft_alpaca.jsonl 37.3 KB jsonl download
_work/curated_harmful/checksums.txt 332 B txt download
_work/curated_harmful/dataset_card.md 1.9 KB md download
_work/curated_harmful/lexsi_provenance.json 642 B json download
_work/curated_harmful/manifest.json 1.9 KB json download
_work/curated_harmful/README.md 2.4 KB md download
_work/curated_harmful/rejected.jsonl 53.9 KB jsonl download
_work/curated_harmful/sft_alpaca.jsonl 34.2 KB jsonl download
_work/harmful_raw.jsonl 38.7 KB jsonl download
benign.jsonl 43.3 KB jsonl download
gsm8k.jsonl 33.5 KB jsonl download
harmful.jsonl 40.4 KB jsonl download
results/ 15 files 3.3 MB
gguf_sha256.txt 336 B txt download
results.json 1.2 KB json download
results.md 624 B md download
runs/q2_k/benign.json 435.7 KB json download
runs/q2_k/gsm8k.json 231.4 KB json download
runs/q2_k/harmful.json 365.1 KB json download
runs/q2_k/lexsi_provenance.json 382 B json download
runs/q2_k/llama-server.log 613.8 KB log download
runs/q2_k/summary.json 483 B json download
runs/q4_k_m/benign.json 446.3 KB json download
runs/q4_k_m/gsm8k.json 209.5 KB json download
runs/q4_k_m/harmful.json 341.6 KB json download
runs/q4_k_m/lexsi_provenance.json 386 B json download
runs/q4_k_m/llama-server.log 611.3 KB log download
runs/q4_k_m/summary.json 477 B json download

README current version from Hugging Face

Shipped ≠ Safe: eval sets and full run logs

Lexsi AI Safety & Alignment Hackathon, Track 2a. Does Tiny Aya Global's safety behaviour survive the
low-bit GGUF files people actually deploy?

Contents

  • eval/harmful.jsonl: 249 prompts a safe model should refuse (JailbreakBench JBB-Behaviors harmful, MIT; XSTest contrast/unsafe, CC-BY-4.0)
  • eval/benign.jsonl: 289 borderline-but-harmless prompts; refusing them is over-refusal (XSTest safe, CC-BY-4.0; JBB-Behaviors benign, MIT)
  • eval/gsm8k.jsonl: 100 GSM8K test questions (MIT), capability check
  • eval/_work/curated_*: CuratorKIT outputs (clean, exact + MinHash dedup, schema gate) with manifest, checksums, provenance and rejected rows
  • results/runs/<variant>/<set>.json: AuditKIT RunResults with every generation and per-sample scores
  • results/results.md / results.json: paired-bootstrap deltas vs the official Q4_K_M with 95% CIs

Prompts include requests for harmful content (from the public benchmarks above) and are for safety evaluation only.
Code: https://github.com/Pavilion-devs/shipped-not-safe

README history 8 revisions

Every night we snapshot the README of every dataset in the catalog. When the SHA changes we archive the new version and diff it against the last. This is the evolving thought record of the alignment-data field: what the author decided to say about the corpus, and how that framing shifted over time.

  1. 2026-11-16 Q2_K GSM8K run + final results table 3a0d569 1.3 KB
    Loading...
  2. 2026-11-16 Q2_K benign-set run 59b5aa0 1.3 KB
    Loading...
  3. 2026-11-16 Q2_K harmful-set run bc7b19c 1.3 KB
    Loading...
  4. 2026-11-16 Upload README.md with huggingface_hub 50d5193 1.3 KB
    Loading...
  5. 2026-11-16 Q4_K_M reference results 507ebb6 1.3 KB
    Loading...
  6. 2026-11-16 Upload README.md with huggingface_hub d3b1cb7 1.3 KB
    Loading...
  7. 2026-11-16 Curated eval sets (CuratorKIT) + provenance 72c1abc 0 B
    Loading...
  8. 2026-11-16 initial commit c9a3289 0 B
    Loading...
Download from HuggingFace

olathepavilion/shipped-not-safe-eval

Catalog is the map. Apps are the tools.

Run models on your own machine, not in the cloud.

Every model page has an "Open in Abliteration" button that hands the model directly to the first-party desktop client, at the quantization your rig can actually run. No API keys, no subscription, no prompt leakage.

Open in Abliteration