language:
- en
license: apache-2.0
tags: - llama.cpp
- gguf
- quantization
- sub-2bpw
- recipe-b
- miniquant
- qwen35
- qad
base_model: huihui-ai/Huihui-Qwen3.5-4B-abliterated
Huihui-Qwen3.5-4B-abliterated-miniQuant (Recipe B)
Sub-2.0 BPW miniQuant Recipe B GGUF quantization of huihui-ai/Huihui-Qwen3.5-4B-abliterated, built and served with fencerJP/llama.cpp-miniQuant.
Available Checkpoints & Benchmark Results
| File | Recipe / Variant | Size | Effective BPW | Prefill Speed (CPU) | Decode Speed (CPU) | Notes |
|---|---|---|---|---|---|---|
Huihui-Qwen3.5-4B-ab_QAT-RecipeB-Distilled.gguf |
Recipe_B (IQ1_S + IQ4_K embd + imatrix + LMS) |
1.05 GiB (1,128.69 MB) | 2.251 bpw | 80.7 – 210.2 tok/s | 26.7 – 28.9 tok/s | Anchored < 2.5 bpw ceiling |
Huihui-Qwen3.5-4B-ab_QAT-Sub2Bpw-Direct.gguf |
Recipe_B Direct (IQ1_S + IQ2_K embd) |
0.95 GiB (967.89 MB) | 1.931 bpw | 213.4 tok/s | 29.5 tok/s | Strictly < 2.0 bpw, 100% coherence |
Creation Process (QAD to Recipe B)
These checkpoints were created using the miniQuant Recipe B sub-2.0 bpw pipeline:
- QAT / QAD Foundation: Starting from Quantization-Aware Training / Distillation (FraQAT + HESTIA bridged-KL distillation) checkpoints in BF16.
- Commutative Rotation Partitioning: 256-dim paired Hadamard rotations applied strictly to contraction down-projections (
ffn_down,ssm_out,attn_output), preserving Gated DeltaNet recurrent state projections (ssm_in,conv1d,dt,alpha,beta) and embeddings in their natural coordinate basis. - Non-Uniform Multi-Bit Sparing (
llama-quantize --recipe-b):- Bulk Trunk (
ffn_up,ffn_gate,attn_qkv,ssm_in):IQ1_S(~1.56 bpw) - FFN Down Projections (
ffn_down):IQ2_XXS(~2.06 bpw) - Attention/SSM Outputs & Values (
attn_output,ssm_out,attn_v):IQ3_XXS(~3.06 bpw) - Boundary Anchors (
token_embd.weight,output.weight):IQ2_Kon 27B/40B (where embeddings are <5% of weights), orIQ4_Kon 4B (where the 248k-vocab embedding table is 15.1% of weights).
- Bulk Trunk (
- Closed-Form LMS Scale Distillation: Out-of-core Least Mean Squares block scale ($d$) refinement against the unquantized BF16 teacher weights.
Hardware Specifications (Creation & Benchmarking)
- CPU: AMD Ryzen AI 9 HX 470 w/ Radeon 890M (12 Cores / 24 Threads, AVX-512 / AVX-VNNI / AVX-512 BF16)
- System RAM: 121 GiB DDR5
- GPU: NVIDIA Tesla V100-SXM2-16GB (16 GiB VRAM)
- Inference / Quantization Engine: fencerJP/llama.cpp-miniQuant (
master/recipe-b)
Usage (llama.cpp-miniQuant)
./build/bin/llama-cli \
-m Huihui-Qwen3.5-4B-ab_QAT-RecipeB-Distilled.gguf \
-p "<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n" \
-n 128 -t 16 --temp 0.0 --repeat-penalty 1.15