license: apache-2.0
library_name: transformers
base_model: Qwen/Qwen3.8-27B
pipeline_tag: text-generation
tags:

• qwen
• abliteration
• abliterix
• lora-self-distillation

Qwen3.8-27B Abliteratex

A refusal-suppressed derivative of Qwen/Qwen3.8-27B, produced with a 10-round iterative LoRA self-distillation pipeline built on Abliterix. The checkpoint is a full BF16 merge and requires no adapter.

Release update — August 20, 2026: this repository now hosts the stronger Abliteratex checkpoint previously released as wangzhang/Qwen3.8-27B-abliteratex. It replaces the earlier two-pass directional-ablation checkpoint. The previous release remains available in this repository's revision history.

⚠️ Responsible-use notice. This model's refusal behavior has been substantially reduced. It will attempt to answer harmful, unethical, or dangerous requests far more readily than the base model. You are solely responsible for how you use it and for complying with all applicable law. Intended for safety research, red-teaming, and evaluation.

Why this model doesn't use classical directional ablation

Qwen3.8-27B's refusals are re-derived during generation rather than encoded at a single residual-stream direction present at the prompt's last token — the same failure mode seen on gpt-oss and VibeThinker-style policy-reasoning models. Classical single-direction ablation was tested and ruled out on this model before this recipe was used:

• A prompt-final-token direction ablates a harmful-topic detector (Cohen's d = 5.4) — cheap to remove, but changes no refusal behavior (100/100 refusals at every KL ≤ 0.01 across 3 independent runs).
• A response-position direction (extracted after an affirmative prefix) has no usable dose-response window: weight 0.5 → KL 0.003 → still 100/100 refusals; weight 1.0 → KL 0.04 → still 100/100; weight 1.5 → KL 1.1 (model destroyed).

Method: iterative LoRA self-distillation

Because the refusal isn't encoded at a fixed direction, this model was produced by teaching the model to imitate its own successful compliant completions rather than by editing weights directionally:

1. Rejection-sample a teacher set. For each of 800 harmful training prompts, generate greedily; for any prompt that still refuses, resample at higher temperature (up to 8 attempts) until a compliant completion is found or attempts are exhausted.
2. Filter for substance. Keep only completions that are non-degenerate, non-keyword-refusal, ≥ 45 words, and ≥ 3 concrete steps — this removes soft-refusal contamination (the model "agreeing" then producing vacuous or off-topic text).
3. Merge with the prior round's filtered teacher set, growing and refreshing the training pool each round.
4. Train a rank-32 LoRA (q_proj, k_proj, v_proj, o_proj, down_proj) via cross-entropy on the compliant continuations, anchored by a benign-prompt KL term (kl_weight = 20) against the frozen base model so behavior on ordinary prompts stays essentially unchanged.
5. Evaluate at several LoRA scales, re-rank the round by refusal count subject to the KL ceiling, and use the strongest LoRA as next round's rejection-sampling teacher.

This loop ran for 10 rounds. Refusal count fell round over round with diminishing returns and one plateau/regression at the end:

R9 at LoRA scale 1.3 is the best point found across all 10 rounds and is what's merged into this repo. R10 repeated the same recipe (teacher = R9) and did not improve on it, so the run was stopped and R9 was shipped rather than continuing to chase the original <10/100 stretch target.

Round-9 adapter hyperparameters

Evaluation

100 held-out harmful prompts and 100 held-out benign prompts (train[800:900], disjoint from the 800 training-time prompts). Refusals judged by google/gemini-3-flash-preview; KL is mean-per-token KL divergence from the base model's output distribution over the benign set.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "wangzhang/Qwen3.8-27B-abliterated"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [{"role": "user", "content": "Your prompt here"}]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    enable_thinking=False,
).to(model.device)

output = model.generate(inputs, max_new_tokens=512, temperature=0.7, do_sample=True)
print(tokenizer.decode(output[0][inputs.shape[-1]:], skip_special_tokens=True))

Intended use & limitations

• Intended for: safety research, red-teaming, robustness/alignment evaluation, and studying refusal mechanisms in policy-reasoning LLMs.
• Not intended for: producing harmful content or any unlawful purpose.
• This is a behavioral fine-tune (LoRA merged into weights), not a knowledge edit — factual accuracy, reasoning, and multilingual ability are inherited from the base model, and the benign-KL anchor was specifically used to keep ordinary-prompt behavior close to base.
• Residual refusals remain (19/100 on the held-out set); this is the strongest point found in a 10-round search, not a guaranteed floor, and behavior may vary outside the evaluated prompt distribution.

Relation to the previous release

The checkpoint that originally occupied this repository used classical two-pass directional ablation (14/100 refusals @ incremental KL 0.0091 vs. pass-1). That method doesn't work well on this model's policy-reasoning refusal (see above); this LoRA-self-distillation approach was built specifically to address that gap on a held-out harmful/benign split (train[800:900]) distinct from the one used for the previous release, so the two refusal numbers are not directly comparable. The previous checkpoint is preserved at revision 0512fe5.

Acknowledgments & citation

• Base model: Qwen3.8-27B (Qwen team).
• Tooling: abliterix.

@software{abliterix,
  title  = {abliterix: automated abliteration of large language models},
  author = {Wu, Steve},
  url    = {https://github.com/wuwangzhang1216/abliterix}
}

License

Released under the base model's Apache-2.0 license.