base_model: LuffyTheFox/Qwen3.8-27B-Uncensored-Genesis-V1-MTP-GGUF
library_name: gguf
pipeline_tag: text-generation
license: apache-2.0
tags:

• gguf
• nvfp4
• fp4
• blackwell
• llama.cpp
• qwen3.8
• uncensored
  language:
• en
• zh

Qwen3.8-27B-Uncensored-Genesis-V1-MTP — NVFP4 GGUF

An NVFP4 quantisation of
LuffyTheFox/Qwen3.8-27B-Uncensored-Genesis-V1-MTP-GGUF,
built to Luffy's requested per-tensor quant policy and verified running on an
RTX 5070 Ti (Blackwell) with native FP4 support enabled.

@LuffyTheFox — this is yours, take it. Mirror it into your repo, or say the word
and I'll open a PR against it directly.

It was actually run, not just produced

Full offload on a 16 GB RTX 5070 Ti, llama.cpp build b10666:

system_info: CUDA : ARCHS = 750,800,860,890,900,1200,1210 | BLACKWELL_NATIVE_FP4 = 1
load_tensors: offloaded 66/66 layers to GPU
load_tensors:   CPU_Mapped model buffer size =   682.03 MiB
load_tensors:        CUDA0 model buffer size = 14114.16 MiB
ftype      : NVFP4

At -c 4096, fully offloaded: ~285–340 t/s prompt, ~42–46 t/s generation.
Output is coherent, the thinking block behaves, and generation stops on a real
end token rather than running to the token limit.

Quant policy

Luffy's policy was written for the MoE Qwen 3.6 35B, which has
ffn_down_exps. This model is dense, so that tensor does not exist here;
the equivalent is ffn_down. Applied that way, every line resolves:

Verified in the finished file — all four F16 tensors are exactly the intended
ones, and the three explicit NVFP4 tensors are NVFP4.

One caveat worth stating plainly: blk.0.attn_gate / attn_qkv were q8_0 in
the source, so F16 there stores dequantised q8 values. It does the intended
job (those tensors keep q8-level fidelity instead of dropping to 4 bits) but it
is not true F16 precision, and it costs more bytes than the source did.

How it was made

The interesting part: no code changes and no rebuild were needed. A stock
llama-quantize can already do this.

llama-quantize does not expose NVFP4 as a base CLI ftype — QUANT_OPTIONS in
tools/quantize/quantize.cpp has no entry for it, and llama-quant.cpp has no
ftype→type case. The obvious read is that you must patch and rebuild. You don't,
because of three things in the existing code:

1. --tensor-type resolves its type argument through ggml_type_name() over
   all ggml types, so =nvfp4 already parses.
2. The override list is vector<pair<regex, ggml_type>> and is matched
   first-match-wins, so a trailing .* catch-all is expressible.
3. Overrides apply whenever the base ftype is quantised
   (ggml_is_quantized(default_type)), so any quantised base works — the
   catch-all then replaces it everywhere.

So the base ftype is a placeholder that never survives, and the real policy is
the regex list. Specific patterns first, catch-all last:

llama-quantize \
  --allow-requantize \
  --tensor-type "blk\.0\.attn_gate\.weight=f16" \
  --tensor-type "blk\.0\.attn_qkv\.weight=f16" \
  --tensor-type "blk\.0\.ffn_down\.weight=f16" \
  --tensor-type "blk\.13\.ffn_down\.weight=f16" \
  --tensor-type "blk\.0\.ssm_out\.weight=nvfp4" \
  --tensor-type "blk\.1\.attn_gate\.weight=nvfp4" \
  --tensor-type "blk\.1\.attn_qkv\.weight=nvfp4" \
  --tensor-type ".*=nvfp4" \
  Qwen3.8-27B-Uncensored-Genesis-V1-Q8_K_P.gguf \
  Qwen3.8-27B-Uncensored-Genesis-V1-NVFP4.gguf \
  Q8_0

Then the file type label, which is the one thing the above gets wrong. The
header inherits the placeholder base ftype and reports file type = Q8_0. That
is cosmetic — loading dispatches on per-tensor types — but it is wrong on the
shelf, so patch the single uint32 in place (no 15 GB rewrite):

python gguf-py/gguf/scripts/gguf_set_metadata.py model.gguf general.file_type 39 --force

39 is LLAMA_FTYPE_MOSTLY_NVFP4. The loader guesses the ftype correctly from
the majority tensor type on its own (GGML_TYPE_NVFP4 → LLAMA_FTYPE_MOSTLY_NVFP4
is already in that switch in llama-model-loader.cpp) and only overrides the
guess when general.file_type is present — so a stale value is actively worse
than none. After the patch it reads ftype : NVFP4.

Notes on the process

• Requantised from Q8_K_P, not from BF16 — no BF16 source was available.
  It is lossy-on-lossy for the tensors that were genuinely q8, though a
  substantial number of source tensors are bf16 and requantise cleanly.
• Luffy's writeup describes dequantising Q8_K_P to FP16 first, then
  requantising. That intermediate step is unnecessary and slightly harmful:
  --allow-requantize already dequantises each block to f32 before
  quantising, so routing through F16 only adds a rounding step.
• --dry-run resolves the whole policy without writing anything, which is how
  the missing ffn_down_exps was caught before spending any time.
• tensor_type_fallback handles tensors whose shape isn't divisible by the
  NVFP4 block size, warning per tensor. Here it never fired.
• The quantise itself took 59 seconds on CPU.

Running it

llama-cli -m Qwen3.8-27B-Uncensored-Genesis-V1-NVFP4.gguf \
  -ngl 99 -c 4096 --temp 0.6 --top-k 20 --top-p 1.0 --min-p 0.0

Sampling above follows the Genesis guidance (temp 0.6, top_k 20, top_p and min_p
disabled). For code, also set --repeat-penalty 1.0: code repeats
identifiers by nature, and any penalty pushes the model off the identifier it
just chose, which is a reliable way to get generated code that won't run.

Needs a Blackwell card and a llama.cpp built with FP4 support for the fast path
(BLACKWELL_NATIVE_FP4 = 1 in system_info); b10666 or newer.

Credits

Base model and quant policy: LuffyTheFox.
Genesis V1 finetune: HauhauCS. This repo is only the NVFP4 conversion and the
verification that it runs.