license: apache-2.0
base_model: JonathanColetti/Qwen3.8-27B-Uncensored
base_model_relation: quantized
library_name: transformers
pipeline_tag: image-text-to-text
tags:

• nvfp4
• compressed-tensors
• quantized
• w4a16
• vllm
• blackwell
  language:
• en
• zh

Qwen3.8-27B-Uncensored — NVFP4A16 (W4A16)

NVFP4 weight-only quantization of
JonathanColetti/Qwen3.8-27B-Uncensored.

51.75 GiB → 19.15 GiB (2.70× smaller).

Read this before deploying: this checkpoint is for Blackwell (SM100/SM120).
NVFP4 GEMMs execute on Blackwell tensor cores. It was produced on an H100
(SM 9.0), which has no NVFP4 hardware path. Benchmarked on that H100 it is
slower than the bf16 original and uses more peak VRAM — see
Hopper benchmark.

Format

Effective ~4.5 bits/weight (4 + 8/16). The block-16 FP8 scale is what
distinguishes NVFP4 from MXFP4, which uses block-32 with a power-of-two E8M0
scale — the finer, non-power-of-two scale retains noticeably more accuracy.

What was quantized

496 Linear modules — 24.351 B params, 87.7% of the model.

The vision tower, lm_head, and embeddings are held at bf16 deliberately —
together only 12.3% of parameters, but the usual accuracy casualties of 4-bit.
The MTP head ships unmodified as model-mtp.safetensors, so speculative
decoding is preserved.

A_log, dt_bias, conv1d, and all norms are not nn.Linear and remain
untouched, which matters here: the model keeps its SSM state in fp32
(mamba_ssm_dtype: float32).

Verified at the tensor level: 496 weight_packed / 496 weight_scale
(F8_E4M3) / 496 weight_global_scale (F32), 0 in model.visual.*.

Memory

Max context is 262,144 tokens. Only 16 of 64 layers use full attention;
the other 48 are linear-attention layers whose recurrent state is constant in
sequence length.

KV cache        64 KiB/token   (16 full-attn layers, GQA 4 kv-heads x 256 head_dim)
KV @ 256K       16.00 GiB
linear state     0.141 GiB     <- constant, at 1 token or at 262,144
                (all 64 layers as full attention would be 64.0 GiB)

Peak at full 256K context, single sequence:

An FP8 KV cache halves the 16 GiB and takes you to ~5–6 concurrent 256K
sequences.

Measured on H100 (SM 9.0) — not the target hardware

transformers reference path, greedy, 40 tokens, causal-conv1d /
flash-linear-attention fast path absent in both runs:

Without NVFP4 tensor cores the runtime unpacks weights to bf16 while still
holding the packed copies — hence higher peak VRAM and ~2× slower than bf16.
Disk and load-time savings are real; inference savings are not. On Blackwell
this table should invert. vLLM has quantized kernels transformers lacks and may
behave differently on Hopper — benchmark your own serving stack rather than
extrapolating from the above.

Reproducing

from transformers import AutoModelForImageTextToText
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
import torch

model = AutoModelForImageTextToText.from_pretrained(
    "JonathanColetti/Qwen3.8-27B-Uncensored",
    dtype=torch.bfloat16, device_map="cuda:0",
)
recipe = QuantizationModifier(
    targets="Linear", scheme="NVFP4A16",
    # NOTE: entries are matched as LITERAL module names unless prefixed "re:".
    # Without the prefix these regexes match nothing and the vision tower is
    # quantized anyway -- while still exiting 0 with a valid-looking checkpoint.
    ignore=["lm_head", r"re:model\.visual\..*", r"re:mtp\..*", r"re:.*embed_tokens.*"],
)
oneshot(model=model, recipe=recipe, output_dir="out", save_compressed=True)

Data-free RTN — no calibration set. NVFP4's global scale derives from the
weights themselves, and group_size 16 is fine-grained enough that RTN holds up.
This also avoids a calibration-distribution mismatch against a fine-tuned model,
and avoids pushing calibration batches through a hybrid linear-attention stack
that llm-compressor has not been validated against.

model-mtp.safetensors must be copied through manually: transformers 5.14.1
does not instantiate the MTP module, so save_pretrained silently drops it.

Produced with llmcompressor 0.13.0, compressed-tensors 0.18.0,
transformers 5.14.1, torch 2.13.0+cu130.

Caveats

• Not validated on target hardware. Produced and tested on Hopper only.
  Blackwell's NVFP4 GEMM has its own accumulation behavior; numbers here are
  indicative, not identical.
• Long-context accuracy is unmeasured. 48 of 64 layers are recurrent, so
  quantization error can compound along the sequence rather than staying bounded
  per token as in softmax attention. Published NVFP4 recipes were validated on
  pure transformers. Evaluate at 2K / 32K / 128K — a single short-context
  perplexity number will not surface this.
• No accuracy evaluation has been run. Only format correctness and a smoke
  generation.
• Quantization perturbs model behavior generally, including tone and refusal
  patterns. This inherits the base model's behavior, which is an abliterated
  fine-tune with no safety filtering; nothing here adds or restores any.
• transformers_version reads 5.14.1 (the tooling pin) vs 5.15.0 upstream.
  A config round-trip diff confirmed zero dropped or altered keys otherwise.