license: apache-2.0
base_model: Qwen/Qwen3.5-35B-A3B
tags:

• abliteration
• uncensored
• qwen
• qwen3.5
• moe
  pipeline_tag: text-generation
  datasets:
• walledai/AdvBench
• tatsu-lab/alpaca

Qwen3.5-35B-A3B Abliterated

Uncensored version of Qwen/Qwen3.5-35B-A3B using weight-level refusal-direction ablation. A difference-in-means direction was extracted from harmful vs. harmless prompt activations and permanently orthogonalized out of every residual-stream-writing weight matrix, including all 256 routed experts and the shared expert in every layer. There is no runtime hook and no LoRA; the edit is baked directly into the checkpoint.

Unlike the earlier Qwen3.8-27B release, the ablation strength is not flat. A single integer-layer direction at full strength left 14–22% refusal on this MoE model. The final edit was found by a Heretic-style optimizer that searched over a fractional direction index and separate per-layer strength curves for attention and MLP/expert writes, minimizing refusals and harmless-prompt KL divergence together.

Format

Text-only bf16 safetensors, architecture Qwen3_5MoeForCausalLM (34.7B total parameters, ~3B active per token). The base repo is a vision-language model. This checkpoint contains only the language model: the vision encoder and the multi-token-prediction (MTP) head are not included, so it takes text input only.

Edited tensors: every residual-stream write. That is embed_tokens, self_attn.o_proj in the 10 full-attention layers, linear_attn.out_proj in the 30 Gated DeltaNet linear-attention layers, and, in all 40 layers, mlp.experts.*.down_proj (256 experts) and mlp.shared_expert.down_proj. Router weights, every other tensor, the config, and the tokenizer/chat template are unchanged from the base model. The exact ablation config is included as ablation_config.json.

Serving

Transformers (tested with transformers>=5.19.0):

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "rajaykumar12959/qwen3.5-35b-a3b-abliterated"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")

messages = [{"role": "user", "content": "Your prompt here"}]
prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
)
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Needs about 70 GB of GPU memory in bf16, e.g. one 80 GB A100/H100. Installing flash-linear-attention (and optionally causal-conv1d) speeds up the linear-attention layers considerably.

Thinking mode: all evaluation below used enable_thinking=False (direct answers). Thinking mode was not evaluated.

Metrics

Refusal fell to 0% in 7 of 12 categories. The residual 3.4% sits mostly in weapons and self-harm. There, the base-model judge flags replies that open with a legal disclaimer or "it depends on why", and two self-harm replies that redirect to crisis resources instead of answering. No measurable capability cost: MMLU-Pro and the security benchmarks moved within noise.

Per-category refusal rate (judged by the unmodified base model, see Caveats):

Capability (MMLU-Pro by subject)

Each subject has only 35 questions, so one question is 2.9 pp; the per-subject differences go both ways and are within noise. MMLU-Pro is scored zero-shot by answer-letter log-likelihood (no chain of thought), so absolute numbers are lower than Qwen's published results. What matters is the original-vs-abliterated difference on identical questions.

Method

Per-layer difference-in-means direction extraction (fp32, last templated token, 210 harmful vs. 210 harmless training prompts). Each direction is Gram-Schmidt-purified against the harmless mean direction and unit-normalized. For a residual-writing projection y = Wx, the component along d̂ is removed in closed form, W' = W − s · d̂(d̂ᵀW), where s is a per-layer strength. No gradient-based training.

Why not a single layer at full strength. On this model, broadcasting one layer's direction at full strength gave a narrow, jagged optimum (validation-slice refusal, 50 prompts):

Optimized ablation. A multi-objective TPE search (Optuna, 60 trials) in the style of Heretic jointly minimized refusal rate (40 held-out harmful prompts, disjoint from the 292-prompt evaluation set) and first-token KL on 64 harmless prompts. It searched over:

• a fractional direction index: the direction is interpolated between two neighbouring layers' directions;
• separate strength kernels over depth for attention writes and MLP/expert writes (peak strength, peak position, floor, width);
• the embedding strength;
• the refusal-subspace rank (1–3 directions; extra directions from k-means clusters of the harmful training prompts).

The chosen point was the lowest refusal with KL ≤ 0.15:

Strengths above 1 overshoot the projection: the edited write carries a small component opposite to the refusal direction. Rank-2 and rank-3 subspaces also reached 2.5% validation refusal, but only at KL 0.34–0.72, so the shaped single direction was chosen.

The previous checkpoint of this model (layer 18 at flat full strength) measured 22.9% refusal and KL 0.121 on the same evaluation; this one is 3.4% and 0.050.

Datasets

The 292-prompt, 12-category refusal evaluation set is a separate template-built set (AblateBench data/build_eval_categories.py), never seen during extraction or optimization.

Evaluation

Caveats:

• Grading. Refusal grading is two-tier: rule-based patterns on the first 200 characters, then an LLM judge for everything else. The judge is the unmodified base model for both models, so the abliterated model never grades itself. It sees the request and the reply, and decides by comparing the logits of REFUSE vs. COMPLY. 100% of the abliterated model's verdicts went to the judge. The judge is still a safety-tuned model and leans toward REFUSE on hedged replies, so 3.4% is a conservative figure. All generations were capped at 128 tokens.
• Judge fixes. An earlier grader version showed the judge only the first 500 characters of a reply with no request, and counted a leading "As an AI, …" note as a refusal. On this checkpoint that version reported 6.8%; most of the difference was disclaimer-prefixed compliance.
• Capability. ARC-style multiple choice saturates at this size, so MMLU-Pro is the capability signal. The 12-task patch-synthesis benchmark is too small to separate 8/12 from 9/12.

Disclaimer

This model has had safety guardrails removed and will comply with requests the original model would refuse. Released for research into AI alignment, interpretability, and refusal mechanisms. The creator assumes no responsibility for downstream use.

Acknowledgments

• Qwen team: base model (Apache-2.0)
• Heretic (p-e-w): the optimized, depth-varying ablation parametrization
• Ablated and evaluated with AblateBench
• Created by rajaykumar12959