license: apache-2.0
base_model:
- llmfan46/Qwen3.6-27B-uncensored-heretic-v2-Native-MTP-Preserved-NVFP4
- Qwen/Qwen3.6-27B
language: - en
pipeline_tag: text-generation
tags: - nvfp4
- vllm
- gb10
- dgx-spark
- sm121
- mtp
- speculative-decoding
- uncensored
- qwen3.6
Qwen3.6-27B-uncensored-heretic-v2-NVFP4-lmheadW4
What this is: the llmfan46 heretic-v2 NVFP4 uncensored checkpoint with a single, high-impact optimization: the lm_head re-quantized from BF16 to NVFP4 W4A16.
Community uncensored/abliterated checkpoints almost always ship the lm_head in BF16 (here ~2.5 GB over a 248,320-token vocabulary). That tensor is read every decode step, so on a memory-bandwidth-bound GPU it dominates the per-token cost and makes decode slow. NVIDIA's own NVFP4 checkpoints quantize the lm_head to 4-bit — which is why they decode fast. We simply brought this checkpoint in line.
Result (measured on NVIDIA GB10 / DGX Spark, sm_121)
| metric | heretic-v2 (BF16 lm_head) | this (NVFP4 lm_head) |
|---|---|---|
| decode tok/s @ 8K (single stream) | ~28 | 41.2 |
| decode tok/s @ 16K | — | 40.6 |
| decode tok/s @ 32K | — | 39.5 |
| prefill tok/s @ 8K | ~2100 | 2155 |
+~47% decode, no quality regression (needle-in-haystack 8K/32K HIT, coherence preserved), uncensored behaviour preserved. On our cluster it now decodes faster than the dense Qwen3.6-27B (~35 tok/s).
How it was made
Standalone lm_head quantization (loads only the lm_head tensor, ~21 GB peak — avoids OOM on the 120 GB unified memory) with NVIDIA modelopt NVFP4QTensor.quantize(W, block_size=16), producing the modelopt tensor layout (weight packed uint8 [V, H/2], weight_scale FP8-E4M3 [V, H/16], weight_scale_2 FP32, input_scale FP32). The format is verified bit-for-bit against a reference NVFP4 lm_head before the shard is rewritten, and lm_head is removed from quantization_config.ignore. Everything else (MoE/attention weights, native MTP tensors) is untouched. Script: see the companion repo.
Serving (vLLM on GB10, custom sm121 build)
vllm serve pocharlies/Qwen3.6-27B-uncensored-heretic-v2-NVFP4-lmheadW4 \
--served-model-name qwen36-27b-uncensored-nvfp4 \
--max-model-len 229376 \
--kv-cache-dtype fp8 \
--quantization modelopt \
--attention-backend flashinfer \
--max-num-seqs 16 --max-num-batched-tokens 32768 \
--gpu-memory-utilization 0.60 \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
--limit-mm-per-prompt '{"image":64}' \
--trust-remote-code
KV cache nvfp4 is NOT possible on GB10 (FlashInfer requires
sm100f; GB10 issm121) → use--kv-cache-dtype fp8.
Attribution & license
Apache-2.0. Derivative of llmfan46/Qwen3.6-27B-uncensored-heretic-v2-Native-MTP-Preserved-NVFP4, itself derived from Qwen/Qwen3.6-27B. Only the lm_head tensor was re-quantized; all other weights are unchanged. Abliteration/uncensoring was performed upstream by llmfan46 — this repo adds only a quantization/performance optimization.