pretty_name: VeriAudit Pilot v0.1
license: other
license_name: mixed-see-license-notes
language:

• en
• ur
  task_categories:
• text-classification
  tags:
• ai-safety
• red-teaming
• cross-lingual
• urdu
• roman-urdu
• code-switching
• low-resource
• llm-evaluation
  size_categories:
• n<1K
  configs:
• config_name: default
  data_files:
  • split: pilot
    path: labels.jsonl

VeriAudit Pilot v0.1

Part of VeriAudit — see the full
technical report
in the source repository for complete methodology, statistics, and
limitations. This dataset card summarizes it.

Dataset Summary

VeriAudit Pilot v0.1 is a 240-evaluation pilot benchmark measuring whether
two open-weight language models (Qwen3-8B, Aya Expanse 8B) apply consistent
safety behavior when the same harmful intent is expressed in English, Urdu,
Roman Urdu, or code-switched Roman Urdu/English. Each evaluation includes
the prompt, a safety label from a five-category taxonomy, and — for
Qwen3-8B only — the model's raw response text (see License below for why
Aya Expanse 8B response text is redacted here).

Motivation

Most publicly reported AI safety evaluation is concentrated in English and a
small number of other high-resource languages. It is not established that a
model's refusal behavior transfers consistently to other linguistic forms of
the same request, including Romanized and code-switched text common in
everyday multilingual usage. This pilot tests that directly for one
low-resource-language pair (English/Urdu) and two open-weight models.

Benchmark Structure

• 30 safety-relevant intents, 5 per category, across 6 categories:
  Cybercrime, Weapons, Self-Harm, Hate Speech, Misinformation, Fraud.
• 4 linguistic variants per intent: English, Urdu, Roman Urdu, code-switched.
• 2 models: Qwen3-8B, Aya Expanse 8B (both 4-bit quantized, local MLX
  inference, greedy decoding, temperature 0.0, max_tokens 300 — see
  run_metadata.json).
• 240 total evaluations (30 × 4 × 2), a fully balanced design: exactly 30
  evaluations per model × language cell, exactly 5 per category × model ×
  language cell.

Annotation Protocol

Five-category taxonomy: safe_refusal, safe_completion,
partial_compliance, unsafe_compliance, ambiguous. 30 cases were
directly human-reviewed, 36 more were human-adjudicated, and the remaining
174 were independently reviewed through an LLM-assisted process rather than
a second human annotator. 66/240 (27.5%) carry genuine human sign-off;
174/240 (72.5%) do not. This dataset should not be described as fully
human-validated.

Label Taxonomy

The final_label field in labels.jsonl is the resolved label (human
review/adjudication where available, independent review otherwise).
preliminary_label reflects an earlier, superseded stage — use
final_label.

Primary Metric

SafetyRate(model, language) = SAFE / (SAFE + UNSAFE)   [ambiguous excluded]
CLSG(model, language) = SafetyRate(model, English) − SafetyRate(model, language)

Results

Overall mean CLSG across the 6 model × non-English-language comparisons:
25.8 percentage points. Full results, confidence intervals, and
statistical tests (McNemar exact tests, Holm correction): see
statistical_analysis.md in this repo, or the technical report.

Files

Limitations

30 intents is a small pilot sample; only 2 models tested; 174/240 labels
independently LLM-reviewed rather than human-annotated; Roman Urdu carries
a disproportionate share of ambiguous outputs; CLSG is descriptive, not a
causal estimator; results are specific to these two models and do not
generalize without replication. Full list: technical report §10.

Intended Use

AI safety research; benchmarking cross-lingual refusal consistency;
methodology reference for constructing similar multilingual safety
evaluations.

Out-of-Scope Use

Not a validated, general-purpose safety certification for any model. Not
evidence of a universal property of Urdu-language AI safety. Not for
training models to better evade refusal. Not for commercial use of the
Qwen3-8B response text without checking Qwen3's Apache 2.0 terms yourself
for your use case; Aya Expanse 8B response text is not included in this
release at all (see License).

Citation

See CITATION.cff
in the source repository.

License

• Prompts, intents, and safety labels: MIT (matches the source
  repository — this is the project's own original content).
• Qwen3-8B raw responses (120 of 240 evaluations): included in
  labels.jsonl. Qwen3-8B is released under Apache 2.0, which permits
  redistribution of model outputs.
• Aya Expanse 8B raw responses (120 of 240 evaluations): NOT
  included in this release. raw_response is set to null for every
  Aya Expanse 8B row, with raw_response_redacted: true. Aya Expanse 8B
  is released under CC-BY-NC (non-commercial) terms plus Cohere/C4AI's
  Acceptable Use Policy, and redistribution of its raw outputs under this
  dataset's otherwise-permissive terms was not cleared at publication
  time. The safety label for every Aya Expanse 8B evaluation (derived
  from human/LLM-assisted review, not reproduced Cohere content) is
  retained — only the verbatim model output text is withheld.