license: apache-2.0
base_model: Qwen/Qwen3.5-9B
library_name: peft
tags:

• safety
• jailbreak-detection
• harm-classification
• representation-engineering
• repshift
• lora
  language:
• en
  pipeline_tag: text-classification

samarth-repshift-9b-jailbreak

A representation-shift jailbreak detection judge built on Qwen/Qwen3.5-9B,
specialized for detecting real harmful content in adversarially-attacked LLM
responses.

Companion to ArthT/samarth-repshift-9b-v1.
The two variants trade off between two failure modes:

Headline result

On the Active Robustness benchmark (7,454 (prompt, response) pairs from
8 defense methods × 4 attack types applied to Llama-3-8B-Instruct, with
ground-truth labels from a gpt-5.4-mini + grok-4.3 dual-rater panel using
a confident-nonsense-aware rubric), discriminating HARMFUL_REAL from
(CONFIDENT_NONSENSE + REFUSAL):

This judge sits within 0.04 AUROC of the strongest production safety
classifier (qwen3guard-gen-8b) on the adversarial detection task, while
remaining a representation-shift judge (2× inference cost, no learned head,
explainable score).

On the harder HARMFUL_REAL vs CONFIDENT_NONSENSE discrimination subtask
(separating real-harm from plausible-looking fake content), this judge gets
AUROC 0.630 — second only to qwen3guard (0.711) and well ahead of all
other open-source safety judges.

Architecture

Same Δ_harm + Δ_hp_axis representation-shift architecture as the
companion v1 variant:

d_drift   = mean over response tokens of ‖h_with(t) − h_without(t)‖
score_raw = d_drift @ Δ_harm − λ_hp · (d_drift @ Δ_hp_axis)
p_harmful = sigmoid(platt_a · score_raw + platt_b)

The difference vs v1 is in training:

• v1 trained with an external honeypot corpus (4,154 records) → tightens the
  honeypot defense, weakens generalization to adversarial defenses.
• This variant trained without external honeypots → broader generalization
  on real adversarial outputs, less aggressive honeypot defense.

Hyperparameters specific to this checkpoint:

• λ_hp_axis = 2.0 (twice the v1 weight on the hp-axis penalty)
• EMA decay for Δ_hp_axis = 0.999 (very slow, stable direction)
• Phase 1 corpus only (no external honeypot augmentation)
• Other hyperparameters identical to v1 (see Training below)

Training

• Base: Qwen/Qwen3.5-9B + LoRA r=16, α=32, dropout=0.05
• Corpus: 9,873 rows = 2,528 harmful prompts (HarmBench / AdvBench /
  JailbreakBench seed) + 2,528 Qwen3-4B-Instruct refusals (same prompts) +
  497 in-house honeypots + 4,320 gray/borderline records
• Loss: `α·shift_unsafe + β·KL_safe + γ·anchor_honeypot + λ_hp·anchor_gray
  • λ_hp_axis·Δ_hp_axis`
• Margins: m_b=5.0, m_h=15.0, m_b_dir=0.5, m_h_dir=1.5
• Weights: α=0.5, β=0.4, γ=0.9, λ_hp=3.0, λ_hp_axis=2.0
• EMA decays: Δ_harm=0.95, Δ_hp_axis=0.999
• Rep layers: 22-30 of 36 (top 30%)
• Schedule: patience 15, eval every 50 steps, lr 1e-4
• Seed: 42

Recommended hybrid usage

For broad-coverage production safety, pair with harmbench-llama at w=0.6
(see the v1 model card for full sweep). For maximum jailbreak coverage,
pair with qwen3guard-gen-8b — both signals are correlated on real harm,
and the multiplicative form sigmoid(qwen3guard) × sigmoid(this) should
inherit qwen3guard's recall (0.92) with substantially lower false-positive
rate.

Per-dataset scores (this variant, alone)

Inference

Repshift is a 2-pass forward (with and without adapter on the same response).
See judges/adapters/samarth_repshift.py in the
Robust-jailbreak-judges
repo for the reference implementation. meta.json carries delta_harm and
delta_hp_axis EMA directions, plus calibrated Platt parameters a and b.

Limitations

• 2× inference cost (forward with adapter + forward without)
• Higher false-positive rate on confident-nonsense honeypots (~0.76 @ p=0.5
  vs v1's 0.009). If your input stream contains many honeypot-style
  responses, use v1 or a hybrid.
• Trained on the Phase 1 corpus only (no external honeypot augmentation),
  so the honeypot-specific defense is weaker than v1.
• English-only training data.
• LoRA-only; the base Qwen3.5-9B weights must be loaded separately.

Citation

@misc{samarth-repshift-9b-jailbreak,
  title  = {Representation-Shift Judges for Adversarial Jailbreak Detection},
  author = {Singh, Arth and AIM Intelligence},
  year   = {2026},
  note   = {Robust Jailbreak Judges project, repshift Qwen3.5-9B
            λ_hp_axis=2.0 variant, seed 42}
}

⚠️ Correction notice (2026-05-22)

Earlier versions of this card claimed AUROC 0.823 on Active Robustness vs
qwen3guard 0.859. Those numbers were computed with a buggy record-level
join. Corrected numbers (HARMFUL_REAL vs CN+REFUSAL, dual-rater gold labels,
2,177 records):

For Active Robustness style adversarial detection, prefer
samarth-qwen35-9b. This
variant is retained for research reproducibility on the project repo's
internal λ-sweep ablations.