license: apache-2.0
base_model: JonathanColetti/Qwen3.8-27B-Uncensored
base_model_relation: quantized
pipeline_tag: image-text-to-text
library_name: transformers
tags:
- qwen3.8
- nvfp4
- fp8
- mixed-precision
- modelopt
- tensorrt
- vllm
- uncensored
- abliterated
- mtp
- vision
Qwen3.8-27B-Uncensored — UD V3 layout, ModelOpt build
Experimental rebuild. It exists to test one hypothesis, not to be the best checkpoint.
This is the same mixed-precision layer map assoppyleon/Qwen3.8-27B-Uncensored-UD-V3-NVFP4
and the same 512 calibration texts, but quantized with
NVIDIA TensorRT Model Optimizer
instead of llm-compressor. Layout and calibration data are held constant so that the
quantizer is the only variable.
Why
The llm-compressor UD V3 build measures KL 2.400 / top-1 0.465 against its BF16 parent
over 180,000 positions. A ModelOpt build of the same parent that quantizes far more
aggressively (all 256 linear layers at NVFP4) measures KL 0.542 / top-1 0.840 — 4.4x more
faithful. Everything else has been ruled out: the layer layout, the W4A4 activation bit-width,
the vLLM kernel, RMSNorm rounding, weight global scales, and gate/up activation ranges all
match. The remaining measured difference is down_proj activation ranges (llm-compressor's are
up to 2.6x wider in deep layers). This checkpoint isolates that: if it scores like the ModelOpt
build, the UD V3 layout is fine and llm-compressor's calibration was the fault.
Parent
JonathanColetti/Qwen3.8-27B-Uncensored @ 5bb7aa90f0efef548e87005b1fb7658e522b6b7f (BF16, 51.77 GiB)
Layer map
| Precision | Modules | Count |
|---|---|---|
| NVFP4 W4A4, block 16, FP8 E4M3 scales | mlp.(gate|up|down)_proj, language-model layers 0–55 |
168 |
| FP8 W8A8 E4M3 | self_attn.(q|k|v|o)_proj (all layers) |
64 |
| FP8 W8A8 E4M3 | linear_attn.(in_proj_qkv|in_proj_z|out_proj) (all layers) |
144 |
| FP8 W8A8 E4M3 | mlp.(gate|up|down)_proj, layers 56–63 |
24 |
| FP8 W8A8 E4M3 | lm_head |
1 |
| BF16 | embed_tokens, model.visual.* (vision tower), all norms, linear_attn.conv1d, linear_attn.in_proj_a, linear_attn.in_proj_b, mtp.* |
— |
The MTP (multi-token-prediction) head and the vision tower are both preserved in BF16.
Calibration
512 sequences, max 2048 tokens each, rendered exactly as the llm-compressor UD V3 build
rendered them — verified by SHA-256 content hash: all 512 match the reference
calibration set byte for byte.
| Domain | Dataset | Revision | Rows |
|---|---|---|---|
| chat | HuggingFaceH4/ultrachat_200k (train_sft) |
8049631c405ae6576f93f445c6b8166f76f5505a |
170 |
| code | nvidia/OpenCodeInstruct (data/train-00000-of-00050.parquet) |
8f3ba5bafe4d6e8db46082cf7ae6741bc370604d |
170 |
| tool | glaiveai/glaive-function-calling-v2 |
e7f4b6456019f5d8bcb991ef0dd67d8ff23221ac |
172 |
ModelOpt calibration algorithm: max (absolute-max amax over the calibration set — no
percentile clipping, no MSE search, no AWQ/SmoothQuant). This is ModelOpt's default and matches
the ModelOpt reference build.
Differences from the llm-compressor UD V3 build
Both are deliberate; neither changes which modules are quantized.
- FP8 group numerics. UD V3 uses compressed-tensors
FP8_DYNAMIC— per-channel static
weight scales with per-token dynamic activation scales. ModelOpt's mixed-precision export
emits per-tensor static FP8 for both weight and activation, and vLLM'sModelOptMixedPrecisionConfigdispatches everyquant_algo: "FP8"entry toModelOptFp8LinearMethod(per-tensor).FP8_PER_CHANNEL_PER_TOKENis not reachable from the
mixed-precision dispatch in vLLM 0.28.0. - No KV-cache quantization. UD V3 declares an FP8 static KV cache with calibrated
k_scale/v_scale. This build declares none (kv_cache_quant_algo: null), matching the
ModelOpt reference build. Serve with--kv-cache-dtype fp8if you want FP8 KV. - No GPTQ. llm-compressor applied
GPTQModifierwithactorder: static, which uses
calibration Hessians to adjust the rounded weights. ModelOpt PTQ does plain round-to-nearest
on the weights; only the activation scales come from the calibration data.
Serving
Requires a Blackwell GPU (SM100/SM120) for the native NVFP4 GEMM.
vllm serve soppyleon/Qwen3.8-27B-Uncensored-UD-V3-NVFP4-modelopt \
--attention-backend flashinfer
vLLM resolves this checkpoint through ModelOptMixedPrecisionConfig
(quant_algo: "MIXED_PRECISION" + a quantized_layers map), givingModelOptNvFp4LinearMethod on the 168 NVFP4 modules and ModelOptFp8LinearMethod on the 233
FP8 modules.
vLLM resolution verified on an RTX PRO 6000 Blackwell Server Edition (SM120):ModelOptNvFp4LinearMethod on all 112 fused NVFP4 modules (56 x gate_up_proj +
56 x down_proj), kernel FlashInferCutlassNvFp4LinearKernel, zero Marlin fallbacks,lm_head on ModelOptFp8LinearMethod with torch.float8_e4m3fn weights.
transformers cannot load this checkpoint directly — NVFP4 packs two 4-bit values per byte.
Storage
21.81 GiB total, 2169 tensors.
| dtype | tensors | size | what |
|---|---|---|---|
uint8 |
168 | 6.97 GiB | packed NVFP4 weights |
float8_e4m3fn |
401 | 10.77 GiB | 233 FP8 weights + 168 NVFP4 block scale tensors |
bfloat16 |
798 | 4.07 GiB | embeddings, vision tower, norms, conv1d, in_proj_a/b, MTP head |
float32 |
802 | <0.01 GiB | 168 weight_scale_2 + 401 input_scale + 233 FP8 weight_scale |
Provenance files
achieved_layer_map.json — the precision actually assigned to every module carrying a weight
quantizer, dumped from the live ModelOpt model before export.input_amax.json — the calibrated activation amax for all 401 quantized modules. Included
because the down_proj activation range is the specific quantity this rebuild is testing.
Produced with nvfp4/modelopt_quantize.py, ModelOpt 0.46.0, torch 2.13.0+cu130,
transformers 5.15.1, on a single RTX PRO 6000 Blackwell Server Edition.