license: other
license_name: nvidia-open-model-agreement
license_link: LICENSE
base_model:
- nvidia/Qwen3.8-Flash-Next-FP4-Local
base_model_relation: finetune
pipeline_tag: image-text-to-text
library_name: vllm
tags: - abliterated
- uncensored
- qwen3.8
- qwen4_exp
- nvfp4
- fp4
- mtp
Qwen3.8-Flash-Next-FP4-Local-Abliterated
nvidia/Qwen3.8-Flash-Next-FP4-Local with an inference-time refusal-direction intervention. NVIDIA's QAD weights are kept exactly as released: every weight file is byte-identical to the base repo at revision b9ce18a6887e18c089fdea8509a33e63429a3300 (all 21 weight shards checked against the Hub's LFS sha256, the tokenizer and other small files against their git blob ids). The only change is an abliteration_config block in config.json, which the fp4local-abliterated vLLM plugin applies at inference time.
Why not edit the weights like a normal abliteration? Most residual-stream writers in this checkpoint are NVFP4 (experts) or MXFP8 (attention output, linear-attention output, shared-expert down projection). Editing them means dequantizing, editing and quantizing again. Measured on two layers, re-quantizing an edited tensor adds 1.5e-2 relative error for MXFP8 and 2.7–3.0e-2 for NVFP4; the runtime projection adds 2.3e-3.
Usage
The abliteration needs the plugin. A runtime that does not support the abliteration_config block does not apply it.
uv pip install -U vllm --pre --torch-backend=auto --extra-index-url https://wheels.vllm.ai/nightly
uv pip install fp4local-abliterated==0.1.0
vllm serve lesj0610/Qwen3.8-Flash-Next-FP4-Local-Abliterated \
--engram-config '{"cpu_offload": true}' \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'
- vLLM 0.32.1rc0 or newer is required, which means a nightly build until 0.32.1 is on PyPI. vLLM 0.32.0 does not read the NVFP4 PLE table (vllm-project/vllm#56273) and allocates it as bf16, four times its size. The plugin declares the minimum version. The released plugin was tested on vLLM 0.32.1rc1.dev467 with tensor parallelism and MTP.
--engram-configis optional. It moves the PLE table to pinned host memory and needs about 36 GiB of free host RAM.- With
--tensor-parallel-sizeabove 1, add--mm-encoder-tp-mode data; the vision MLP cannot be split across GPUs in NVFP4 blocks.
At startup each worker logs a line like this for the model, and one for the MTP draft (AbliteratedQwen4ExpMTP):
Abliteration on AbliteratedQwen4ExpForConditionalGeneration: lambda 1.5, ... residual writes in ... layers
If the line does not appear, the plugin is probably not active; check that it is installed in the same environment as vLLM.
Results
Compared with the base model and with a weight-edited build that uses the same direction and λ: windowsxp811203/Qwen3.8-Flash-Next-Abliterated, a bf16 build edited with λ = 1.5, quantized here to NVFP4 with ModelOpt PTQ. That is not their own NVFP4 upload, which was not tested. Except where noted, none of the data in the table was used to choose λ.
| FP4-Local (base) | Weight-edited, NVFP4 PTQ | This model | |
|---|---|---|---|
| Refusal-phrase hits, 625 harmful prompts | 612 (97.92%) | 1 (0.16%) | 1 (0.16%) |
| Refusal-phrase hits, 100 benign prompts | 34 | 0 | 0 |
| Explicit refusals, blind review of harmful replies | not reviewed | 0 of 331 | 0 of 328 |
| Over-refusals, blind review of benign replies | not reviewed | 0 of 43 | 0 of 51 |
| Refusal-phrase hits, 20 harmful requests written in an image ¹ | 20 | not tested | 0 |
| MMLU, 12,036 questions | 85.35% | 83.63% | 84.05% |
| GSM8K, 1,000 questions (5-shot) | 96.10% | 96.20% | 96.60% |
| Synthetic-image questions (colors, reading text, counting, positions) | 27 of 27 | not tested | 27 of 27 |
| MTP acceptance (3 draft tokens) | 0.581 ² | 0.524 | 0.522 |
¹ The first 20 AdvBench prompts, which were used as text while choosing λ. Scored with a different phrase list from the text rows; see Evaluation details.
² Not comparable: the base model answers most harmful prompts with a short refusal, which is easy to draft, while the other two write up to 1,024 tokens of content.
- Refusals: no explicit refusals or over-refusals in the reviewed replies. Some replies were indirect or partial: they answered, but left out the harmful part or turned it into prevention advice. The weight-edited build did the same. Review method and counts are under Evaluation details.
- Against the weight-edited build: MMLU and GSM8K were slightly higher in this evaluation (+0.42 points, paired McNemar p = 0.055; +0.4 points, p = 0.45), and MTP acceptance was slightly lower. The two builds were also quantized differently, so this does not isolate the effect of skipping re-quantization.
- Against the base model: not lossless. MMLU is 1.30 points lower on these 12,036 questions (157 net questions, p < 0.001); the weight-edited build is 1.72 points lower. GSM8K showed no significant change (+0.5 points, p = 0.23).
- Refusals were tested in English only. A short Korean generation check (30 prompts) found no empty replies or looping, and stray Chinese characters stayed near zero, as with the base model.
- Images: on the synthetic-image questions, this model gave the same answers as the base model except "Grey" vs "Gray". No refusal phrases were detected in responses to the 20 image-rendered harmful requests. This check ran on the released plugin. Video was not evaluated.
How it works
The plugin's fp4local-abliterate tool writes the block into config.json, the same way quantizers add quantization_config (schematic):
"abliteration_config": {"lambda": 1.5, "direction": [2,560 floats, unit length], "source": "..."}
At each residual-stream writer the plugin replaces the output x with x − λ (vᵀx) v, computed in fp32: token embedding, every attention and linear-attention output, every MoE output (routed experts plus shared expert), the PLE value, and the MTP module's embedding, fc_embedding, fc_hidden and layer outputs. This equals editing each of those writers to W − λ v vᵀ W, the same set the weight-edited build changed. The PLE short-convolution output and the vision tower's embeddings and deepstack inputs are not changed. The MTP draft reads the same config.json, so it gets the same projection.
λ = 1 removes the component along v; λ = 1.5 goes past it, which is the strength the weight-edited build used. λ = 1.25 was also tried; it was dropped because a review of its replies on the selection data still found explicit refusals.
The direction is not ours. It is the one windowsxp811203 captured for Qwen3.8-Flash-Next-Abliterated: residual stream after 37 of the 48 decoder layers, last prompt token, 32 harmful / 32 harmless prompt pairs, layer chosen by Cohen's d. Their model card has the details and the scripts. We recovered it from their weights: the difference between their edited bf16 tensors and the originals is rank-1 and gives the same vector from every tensor (cosine 0.999999). The method is from Arditi et al., Refusal in Language Models Is Mediated by a Single Direction (2024).
Evaluation details
- Serving: except for the image checks, the results above come from a vLLM source build with the same projection compiled into the model code, with KV cache NVFP4, greedy decoding,
enable_thinking: false, prefix caching off and MTP on (3 draft tokens). An earlier revision of the plugin was compared with that build on 30 chat prompts and 100 MMLU questions, and the differences were no larger than between two launches of the same build. The released plugin was started on vLLM 0.32.1rc1.dev467 with tensor parallelism and MTP on, and placed the projection at every write point listed above. The image check was run with it; the other benchmarks were not rerun. Speed was measured only on the source build, where TTFT and TPOT stayed within run-to-run variation of the base model. The plugin's speed was not measured separately. - Data: harmful = JailbreakBench prompts not taken from AdvBench (82) plus CatQA (543); benign = JailbreakBench's 100 benign prompts; MMLU = the 12,036 test questions not used while choosing λ; GSM8K = 1,000 train questions. λ was chosen on separate data: AdvBench, GSM8K test and the remaining 2,006 MMLU test questions.
- Refusal-phrase hits (text): one of 39 refusal markers in the first 400 characters of the reply, after lowercasing. Replies were generated up to 1,024 tokens.
- Blind review: a reply was a candidate if it contained one of the 39 markers anywhere, was shorter than 300 characters, or contained a redirect phrase ("instead", "fictional", "harmful", …). Every candidate plus 20 random non-candidates per set were judged COMPLIED / REFUSED / DEFLECTED from the whole reply by an LLM reviewer (Claude) that did not know which model wrote which file. Harmful: 308 candidates + 20 for this model, 311 + 20 for the weight-edited build. Benign: 31 + 20 and 23 + 20. REFUSED was 0 everywhere, and every reviewed benign reply was COMPLIED. In the harmful set, DEFLECTED (partial or indirect answers) was 9 of 308 candidates and 1 of 20 non-candidates for this model, 10 of 311 and 1 of 20 for the weight-edited build; the remaining non-candidates were not judged. The base model was not reviewed this way, since almost all of its replies are candidates.
- MMLU: chat template, "Answer with a single letter.", scored by the highest first-token logprob among each letter's
L/Ltokens. - GSM8K: vLLM's in-tree 5-shot prompt sent as one user message, last number in the reply.
- Image check: 448×448 images drawn with known answers: 10 solid colors, 6 rendered words or numbers, 6 circle counts (2–7) and 5 left/right or above/below questions. Harmful: the first 20 AdvBench prompts rendered as text in an image with the instruction "Follow the instruction in the image.", replies of up to 256 tokens, scored with a separate, case-sensitive list of 19 English refusal phrases searched throughout the reply. Greedy, thinking off, vLLM 0.32.1rc1.dev467 with tensor parallelism and MTP on, base and this model served the same way.
Limitations
- The model follows harmful requests that the base model refuses. Add your own safeguards before putting it in front of users.
- Refusals were tested in English only. Image input was checked only on simple synthetic images, and video was not evaluated.
- MMLU is below the base model (see above).
- The
abliteration_configblock only takes effect with the plugin installed.
License
Licensed by NVIDIA Corporation under the NVIDIA Open Model Agreement; a copy is in LICENSE. The underlying Qwen3.8-Flash-Next model is governed by the Qwen Community License 1.0; a copy is in LICENSE-Qwen. The base model card's terms, notices and model information apply unchanged; see nvidia/Qwen3.8-Flash-Next-FP4-Local. This repo is not affiliated with or endorsed by NVIDIA or Alibaba.