base_model: local-inference-lab/Qwen3.8-Flash-Next-NVFP4
base_model_relation: finetune
license: other
license_name: qwen-community-1.0
license_link: https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/LICENSE
pipeline_tag: text-generation
library_name: transformers
tags:
- abliterated
- reduced-refusal
- nvfp4
- mxfp8
- modelopt
- quantization-aware-distillation
- conversational
Qwen3.8-Flash-Next-NVFP4-Abliterated
Reduced-refusal model. A refusal-direction edit was applied deliberately. Read the safety section before deploying.
Summary
A refusal-direction projection applied directly to the quantised checkpointlocal-inference-lab/Qwen3.8-Flash-Next-NVFP4, not to its BF16 parent.
That distinction is the point of this build. The parent is not a post-training
quantisation — it is quantisation-aware distilled, 2,500 trunk updates plus
1,500 joint-refinement updates against the BF16 teacher over 200 M tokens.
Abliterating Qwen/Qwen3.8-Flash-Next and re-quantising would throw that
distillation away and produce a different, weaker artefact. Editing the quantised
weights in place is the only way to get an abliterated version of this checkpoint.
No tensor changed format. Every tensor keeps the dtype, block-scale tensors
and byte size it has in the parent. 25188 residual-writing tensors were
edited; the remaining shards are hardlinked from the parent and are byte-identical.
What was edited
| Class | Count | Format | Method |
|---|---|---|---|
Attention output projections (linear_attn.out_proj, self_attn.o_proj) |
48 | MXFP8 | constrained requantisation |
Shared-expert down_proj |
48 | MXFP8 | constrained requantisation |
embed_tokens, ple.value_proj, MTP write projections |
4 | BF16 | exact projection |
Routed-expert down_proj |
25088 | NVFP4 | constrained requantisation |
Direction measured at decoder layer 37 of 48 (Cohen's d = 8.57),
from 128 harmful (mlabonne/harmful_behaviors) and 128 harmless
(mlabonne/harmless_alpaca) prompts, seed 42, as the normalised difference of
means of the residual stream entering that layer. Projection strength alpha = 1.0.
Why a naive edit does not work here
The stored weights already sit exactly on the quantisation grid, so a rank-1
projection worth 2-4 % of a tensor's Frobenius norm is mostly smaller than
half a unit in the last place and plain round-to-nearest returns the original
code. Measured on this checkpoint's own tensors, round-to-nearest retains only
74 % of the edit on MXFP8 and 14 % on NVFP4 — a checkpoint that looks
edited and largely is not.
This build instead chooses, per element, the code that both approximates the
edited weight and holds the constraint v · W = 0 across each column: rows are
visited in descending |v_i| and each absorbs as much of the accumulated
constraint error as a one-ULP move allows.
Residual leakage (||v.W_hat|| / ||v.W||; 0 = applied exactly, 1 = erased) is at
most 8.88e-07 across all 25184 quantised tensors, and
6.87e-03 across the 4 BF16 ones -- the BF16 tensors are the
looser of the two, because they take plain rounding and inherit bfloat16's 0.39 %
relative ULP, while the quantised ones are actively driven to zero. Mean added
weight error is 3.966% of each tensor's norm.
Evaluation
Capability parity against the parent, measured in the same session on the same
serving configuration. Scores from different serving configurations are not
comparable for these profiles.
| Profile | parent | this build |
|---|---|---|
| lavd (ledger consistency), n=10 | 10/10 | 9/10 |
| lavd, n=30 | 29/30 (96.7 %) | 28/30 and 30/30 (58/60 = 96.7 %) |
| estonia (long-context retrieval), n=30 | 30/30 | 30/30 |
lavd is the discriminating profile for this family; estonia is not (every
build measured scores 30/30). At n=10 a single difference is inside binomial
noise, so lavd was re-run at n=30 against the parent — that is the row to read.
One run in 30 hits the 40,000-token generation cap on the parent as well as on
this build, so the occasional runaway is a property of the task, not of the
edit.
Refusal behaviour on treadon/abliteration-eval (333 prompts):
| Split | parent | this build |
|---|---|---|
| harmful — compliance (higher = more abliterated) | 10.50% | 94.50% |
| over_refusal — compliance (higher is better for any model) | 93.98% | 91.57% |
| multilingual — compliance | 90.00% | 94.00% |
Safety
This model's refusal behaviour was deliberately reduced. It will comply with
many requests the parent would decline, and it carries no added safety
mitigations. Refusal rates above are behaviour measurements, not a safety
endorsement. Deploy only behind appropriate policy, filtering, access control and
legal review.
Serving
Same runtime and recipe as the parent — mixed NVFP4/MXFP8 ModelOpt layout,
MTP3, FP8 KV cache, and the n-gram (PLE) table offloaded to host RAM. The
quantisation is unchanged, so anything that serves the parent serves this.
vllm serve huginnfork/Qwen3.8-Flash-Next-NVFP4-Abliterated \
--served-model-name Qwen3.8-Flash-Next \
--quantization modelopt_mixed --kv-cache-dtype fp8 --dtype bfloat16 \
--max-model-len 262144 --max-num-seqs 16 --max-num-batched-tokens 6019 \
--enable-prefix-caching --enable-chunked-prefill --language-model-only \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
VLLM_PLE_CPU_OFFLOAD=1 is required to fit a single 96 GB card.
Provenance
- Parent:
local-inference-lab/Qwen3.8-Flash-Next-NVFP4 - Method: single refusal-direction projection, applied in-format; full per-tensor
record inabliteration_report.jsonin this repo. lm_headreads from the residual stream rather than writing to it and was not
edited. The vision tower was not edited.- Per-run benchmark JSONs for both this build and the parent are under
measurements/in this repo, so every number above can be recomputed.
A more conservative variant — the same method applied to the attention output
projections, shared experts, embeddings and MTP writes only, leaving the 25,088
routed-expert down_proj tensors untouched — was built and measured alongside
this one. It is equally capability-neutral but only reaches 51 % harmful refusal
against this build's 5.5 %, so it was not published.