base_model:
- google/gemma-4-31B-it
tags: - gguf
I've noticed that higher quality for global layers (17% of the model) gives better overall quality than the standard mixed like compression, for models of the same size or 1-2 gigabytes larger.
Better than QAT from google (or unsloth/gemma-4-31B-it-qat) ⸺ The QAT model makes gross factual errors when writing names or exact facts, which neither the original model nor my model does. Probably, q4_0 is much more important for global layers than I thought, and it's broken in QAT models. Perhaps if you replace all the global layers and embeddings with q6_k in the QAT model, it will be better. I'm even curious if the QAT q4_0 local layer will be better in quality than the q4_k layers from the original model. But I won't do such complex merges, too lazy.
I recommend using only with llamacpp b9918 and not newer.
q.bat
"llama-b9016-bin-win-cpu-x64/llama-quantize.exe" ^
--tensor-type "blk\.(5|11|17|23|29|35|41|47|53|59)\.attn_q.weight=q6_k" ^
--tensor-type "blk\.(5|11|17|23|29|35|41|47|53|59)\.attn_k.weight=q6_k" ^
--tensor-type "blk\.(5|11|17|23|29|35|41|47|53|59)\.attn_v.weight=q6_k" ^
--tensor-type "blk\.(5|11|17|23|29|35|41|47|53|59)\.attn_output.weight=q6_k" ^
--tensor-type "blk\.(5|11|17|23|29|35|41|47|53|59)\.ffn_gate.weight=q6_k" ^
--tensor-type "blk\.(5|11|17|23|29|35|41|47|53|59)\.ffn_up.weight=q6_k" ^
--tensor-type "blk\.(5|11|17|23|29|35|41|47|53|59)\.ffn_down.weight=q6_k" ^
--tensor-type "blk\.\d+\.attn_q.weight=q4_k" ^
--tensor-type "blk\.\d+\.attn_k.weight=q4_k" ^
--tensor-type "blk\.\d+\.attn_v.weight=q4_k" ^
--tensor-type "blk\.\d+\.attn_output.weight=q4_k" ^
--tensor-type "blk\.\d+\.ffn_gate.weight=q4_k" ^
--tensor-type "blk\.\d+\.ffn_up.weight=q4_k" ^
--tensor-type "blk\.\d+\.ffn_down.weight=q4_k" ^
--token-embedding-type q6_k ^
gemma-4-31B-it-bf16.gguf gemma-4-31B-it-Q4_K-local-Q6_K-global.gguf Q2_K