license: apache-2.0
base_model:
- LuffyTheFox/Qwen3.6-35B-A3B-Uncensored-Genesis-Hermes-MTP-GGUF
tags: - gguf
- nvfp4
- qwen35moe
- moe
- mtp
- multimodal
Qwen3.6-35B-A3B-Uncensored-Genesis-Hermes-MTP NVFP4 GGUF
NVFP4 requantization of LuffyTheFox/Qwen3.6-35B-A3B-Uncensored-Genesis-Hermes-MTP-GGUF.
Source: Hermes3.6-35B-A3B-Uncensored-Genesis-MTP-Q8_K_P.gguf (44.5 GB, file_type 7 / Q8_K_P).
Files
| File | Size | Notes |
|---|---|---|
Hermes3.6-35B-A3B-Uncensored-Genesis-MTP-NVFP4.gguf |
20.4 GB | file_type=39, inline UE4M3 scales |
mmproj-Hermes3.6-35B-A3B-Uncensored-Genesis-F16.gguf |
0.9 GB | F16 vision projector (unmodified from source repo) |
Architecture
qwen35moe— hybrid MoE: 40 layers (30 DeltaNet/SSM + 10 full-attention, every 4th) + 1 appended MTP/NextN decoder block (blk.40)- 256 experts, 8 active + shared expert (~A3B active params)
nextn_predict_layers=1— MTP draft head preserved; works with--spec-type draft-mtpcontext_length=262144
Quantization recipe
Direct one-step Q8_K_P -> NVFP4 with an advanced llama-quantize build (llama.cpp-derived NVFP4 quantizer):
llama-quantize --allow-requantize --nvfp4-inline-scales-only \
--tensor-type-file tensor_types.txt <Q8_K_P source> <output> NVFP4 6
- Inline NVFP4 scales (UE4M3) — compatible with runtimes that do not support separate
.scale/.input_scaletensors - No imatrix used
- Per-tensor override list (483 rules): norms/SSM state/router tensors at F32, shared-expert
shexp,attn_k/v,ssm_beta,token_embdand singular-collapse-proneblk.0/blk.13tensors at F16,output.weightat Q6_K - The whole MTP block (
blk.40) follows the quantizer's default MTP matrix policy: NVFP4
Tensor mix (753 tensors)
| Type | Count |
|---|---|
| NVFP4 | 237 |
| F32 | 339 |
| F16 | 175 |
| Q6_K | 1 (output.weight) |
| Q8_0 | 1 (MTP router) |
Verified
Tested with upstream llama.cpp llama-server on Ampere (RTX 3090 / 3080 Ti):
- Text: coherent reasoning output, arithmetic correct
- MTP speculative decoding (
--spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.75): draft acceptance ~93% - Vision via the F16 mmproj: image captioning works
Usage
# full GPU offload (needs ~24 GB VRAM incl. mmproj + KV)
llama-server -m Hermes3.6-35B-A3B-Uncensored-Genesis-MTP-NVFP4.gguf \
--mmproj mmproj-Hermes3.6-35B-A3B-Uncensored-Genesis-F16.gguf \
-ngl 999 --flash-attn on \
--spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.75
# smaller GPUs: keep MoE experts in RAM, dense part on GPU (~11 GB VRAM, ~18 GB RAM)
llama-server -m Hermes3.6-35B-A3B-Uncensored-Genesis-MTP-NVFP4.gguf \
--mmproj mmproj-Hermes3.6-35B-A3B-Uncensored-Genesis-F16.gguf \
-ngl 999 --n-cpu-moe 41 -t 8 --flash-attn on \
--spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.75
Requires a recent llama.cpp build (or another runtime with qwen35moe + file_type 39 NVFP4 support).
Credits
- Original model: LuffyTheFox