pretty_name: "Type-Safe Models: Safety Classification and Choice Robustness"
language:

• en
  license: other
  license_name: mixed-source-attribution
  license_link: LICENSE.md
  task_categories:
• text-classification
  tags:
• model-evaluation
• safety
• option-order
• robustness
• jev
  configs:
• config_name: requests
  default: true
  data_files:
  • split: original
    path: data/requests-original.parquet
  • split: reversed
    path: data/requests-reversed.parquet
  • split: serialization_control
    path: data/requests-serialization_control.parquet
• config_name: case_index
  data_files:
  • split: cases
    path: data/case-index.parquet
• config_name: model_results
  data_files:
  • split: original
    path: data/model-results-original.parquet
  • split: reversed
    path: data/model-results-reversed.parquet
  • split: serialization_control
    path: data/model-results-serialization_control.parquet
  • split: repeat
    path: data/model-results-repeat.parquet
  • split: unsupported
    path: data/model-results-unsupported.parquet

Type-Safe Models: Safety Classification and Choice Robustness

Harmful/benign classification and answer-choice robustness across five type-safe models.

We test whether five models correctly classify requests from XSTest, OR-Bench, and HarmBench, and whether reversing the choices changes their decisions. These are classification results, not tests of actual refusal or deployment safety.

The release contains 83,183 source requests in three choice variants (249,549 rows), with instructions, expected answers, source references, scoring tools, and model settings. All five models were evaluated on original and reversed inputs; only Original Jev was evaluated on order-only inputs.

Here, type-safe models means models evaluated through fixed-choice interfaces. “Type-safe” describes the output format, not decision correctness or safety. This grouping does not imply a shared architecture or affiliation with TypeSafe.

Takeaway: Laya and Laya Multilingual had the largest gain (+15 points, copyright validation); Laya Multilingual had the largest drop (−26.3 points, contextual validation). Both occurred on small HarmBench validation sets. CLM’s accuracy did not change.

Blue means higher accuracy; red means lower accuracy. Numbers show reversed minus original accuracy, in percentage points. The −26.3-point drop represents five fewer correct answers among 19 requests.

Laya’s OR-Bench Toxic score stayed the same even though 56 answers changed. The next chart shows whether individual answers stayed the same.

Takeaway: Across the eight groups with the same supported requests, CLM had the highest consistency, sometimes tied. Laya Multilingual was lowest in seven groups; Laya was lowest on HarmBench standard validation. CLM scores choices separately, which is consistent with its stability. Contextual groups use different subsets and are not ranked here.

Consistency compares the answer meanings, not the A/B letters. Counts show same answers / requests answered in both orders. It does not measure correctness. A model that always gives the same answer is perfectly consistent on any benchmark.

What we found

• The same score can hide different decisions. On OR-Bench Toxic, Laya got 273/655 correct (41.7%) in both choice orders, yet 56/655 answers (8.5%) changed: 28 became correct and 28 became incorrect.
• The stronger model depends on which errors matter. With original choices, Original Jev identified 603/655 harmful OR-Bench Toxic requests (92.1%), versus Laya’s 273/655 (41.7%). On benign OR-Bench Hard requests, the ranking reversed: Laya got 1,204/1,319 (91.3%) correct, versus Original Jev’s 447/1,319 (33.9%).
• Sensitivity to reversed choices varied across benchmarks. Reversal changed 150/450 Laya Multilingual answers (33.3%) on XSTest, but 66/80 (82.5%) on HarmBench’s copyright test set. Copyright uses a separate “disallowed reproduction” question; it is not a harmful/benign test.

Our reversal changes both choice order and A/B assignments, so it does not isolate order alone. These observations describe this evaluation; they do not establish general model safety.

What you can download

The dataset contains 83,183 source requests, each in three versions: 249,549 rows total. Every row includes the full request, instructions, answer choices, expected answer, and source reference.

These versions reuse the same requests. They are not separate training and test sets. Match the same request across versions with pair_id.

Per-example results are included: 1,161,150 scored answers plus 352 unsupported records. You can recompute the reported scores from this download.

Join results to requests using pair_id and variant. A result is identified by (model, pair_id, variant, repeat_index); repeat zero is the main evaluation. Repeats are partial and must not inflate primary scores.

from datasets import load_dataset

results = load_dataset("SabaPivot/jev-safety-bench", "model_results", split="original")

Rows include predictions, correctness, available probabilities, model revisions, request hashes, and recorded execution identifiers. Null means unavailable. See RESULTS_SCHEMA.md for field meanings.

To recompute and check all reported results after downloading the repository:

python tools/score_release.py --output /tmp/recomputed-results.json

This checks agreement with the saved aggregate report, without calling models. It verifies the published calculations, not whether a future rerun returns identical answers.

What “reversed” means

Here is an example. The original A/B assignment varies by request.

The reversed test changes both the answer letters and the order of the answer texts. The order-only test keeps each letter's meaning. Only Original Jev was evaluated on the order-only test. An API may reorder choices internally, so this test cannot prove how the model processes their order.

Test your own model

Follow BENCHMARK_GUIDE.md for the steps and prediction file format. Use tools/score_predictions.py to score your answers. CONFIGURATIONS.md gives the exact questions, model formats, and settings we used.

Load directly from Hugging Face:

from datasets import load_dataset

repo = "SabaPivot/jev-safety-bench"
original = load_dataset(repo, "requests", split="original")
reversed_inputs = load_dataset(repo, "requests", split="reversed")
assert original[0]["pair_id"] == reversed_inputs[0]["pair_id"]

For reproducible runs, add the same revision="<commit hash>" to both calls and record it. After downloading the repository, preview one request without running a model:

python tools/render_requests.py data/requests-reversed.parquet --row 0 --model typesafe/jev-1.13-20260917

This command only prints a request. Without --model, the tool uses the rolling alias typesafe/jev-1.13. A new run may resolve to a different model; record the actual returned version. The adapter snapshots are reference code, not a complete runnable environment. See CONFIGURATIONS.md.

Send the model the request (state), instructions, and answer letters/texts in the stored order. Do not send the expected answers (gold_label, gold_key, or semantic_label). Use the format your model expects; see CONFIGURATIONS.md.

Compare what the answers mean, not just their letters. After reversal, “A” can mean the opposite answer.

Which benchmarks are included?

HarmBench includes 320 test and 80 validation requests. We report standard, contextual, and copyright requests separately. Copyright requests have their own instructions and labels. The 110 HarmBench requests that need images are excluded.

All text and context are included in full. Laya could not read 27 long requests; Open-Jev could not read 61. The other three models answered every text request.

Results

We collected 1,161,150 model answers, including repeat tests. The tables below show the results. Per-example answers and probabilities are available in the model_results configuration.

Correct means the model agreed with the benchmark's label. For example, 234 / 250 (93.60%) means the model correctly marked 234 of 250 benign requests as benign. The other 16 were wrongly marked harmful. Skipped requests are shown separately.

Results at a glance

Percent correctly identified, using the original choices. Each column uses only requests with the named benchmark label. Higher is better. HarmBench here is the standard test set; all other sets, exact counts, and errors are folded below.

XSTest detail

Takeaway: Original Jev performed best on both axes: 93.6% benign and 94.0% harmful accuracy. Laya Multilingual was lowest on benign requests (52.0%); Open-Jev was lowest on harmful requests (41.5%).

The scatter plot uses the original choices. Top-right is better at both; counts list benign, then harmful.

Takeaway: Original Jev classified both classes most accurately. Laya Multilingual was lowest on benign requests (130/250); Open-Jev was lowest on harmful requests (83/200).

Takeaway: CLM had the fewest changed answers (0/450); Laya Multilingual had the most (150/450). Fewer changes means greater consistency, not necessarily better classification.

Takeaway: CLM was most consistent: 450/450 answers stayed the same. Laya Multilingual was least consistent: 300/450 stayed the same. This ranks consistency only, not correctness.

Takeaway: Original Jev had the highest benign and harmful accuracy in both choice orders. Laya Multilingual had the lowest benign accuracy, while Open-Jev had the lowest harmful accuracy in both orders.

Related work

• Sun et al. (2026) study meaningful option names and their assigned descriptions. Our A/B reversal changes both assignments and presentation order; it is a different test.
• XSTest and OR-Bench study refusal of benign requests. We adapt their requests into classification questions.
• Arun and Ma’s Cisco study (2026) evaluates Jev and Laya with a different taxonomy and model-generated labels. We use original benchmark labels and matched choice variants.

These are works by other authors. This release adds comparative results and reusable inputs; it does not claim the first discovery of choice sensitivity or Jev safety failures.

How to read these results

• Check both benign and harmful scores. A model that calls everything harmful gets every harmful request right but every benign request wrong.
• OR-Bench's individual sets and HarmBench's categories each have only one expected answer. Always choosing that answer scores 100%. XSTest tests both harmful and benign requests.
• OR-Bench Hard and 80K share 1,318 identical requests. Reversed choices and repeat runs also reuse requests. Do not count these as new, independent examples.
• We reuse source labels without a new annotation study validating this classification wording. Rankings describe these requests and settings; we did not test statistical significance or establish broader model superiority.
• We chose the wording after inspecting preliminary model outputs. This was exploratory, not a preregistered or held-out prompt evaluation. The copyright question is exploratory, and benchmark exposure during model training is unknown.
• CLM scores candidate texts separately. Its unchanged answers under reordering are consistent with that design; this alone does not demonstrate better learned robustness.
• Models use different training, architectures, and input formats. These results do not isolate why a model succeeds or fails. We do not combine them into one “alignment score.”

Models and settings

We tested Original Jev, CLM, Laya, Open-Jev, and Laya Multilingual. Exact versions and settings are in CONFIGURATIONS.md, models.json, and exact-configurations.json.

Original Jev ran through OpenRouter using typesafe/jev-1.13-20260917.

Sources and licenses

The original benchmark authors created the requests used here. Please credit and cite their work when using this dataset.

We adapted their requests into questions with fixed answer choices. These are not the benchmarks' official generation or refusal scores, and their authors have not endorsed this evaluation.

See LICENSE.md and licenses/ for the source terms. Cite the original benchmark papers when using their requests. Exact source versions are in source-manifest.json.

This release adds changed-choice versions, evaluation tools, model settings, and the results above.

Full accuracy scores

Each OR-Bench set and HarmBench category has only one expected label, so a constant classifier can score 100% on an individual column. Read benign and harmful results together. These are the underlying scores for the change chart above. Percentages show correct answers among scored requests. Benign and harmful requests, and HarmBench test and validation sets, stay separate. Copyright uses its own “disallowed reproduction” label. Laya skips 27 long requests; Open-Jev skips 61. OR-Bench Hard overlaps with 80K, so these columns are not independent tests.

Takeaway: Original Jev led both XSTest classes and HarmBench standard/copyright, tying CLM on standard validation. Open-Jev was lowest on harmful XSTest and HarmBench standard/copyright; Laya Multilingual was lowest on benign XSTest. Laya led benign OR-Bench sets, where CLM was lowest. CLM led Toxic, where Laya Multilingual was lowest. Contextual columns are not ranked because coverage differs.

Takeaway: The leaders and lowest scorers remained the same on XSTest, OR-Bench, and HarmBench standard/copyright, except Original Jev now tied CLM on both standard splits. Contextual columns are not ranked because coverage differs; consult “Comparing models on the same requests” for original-choice results on shared inputs.