license: apache-2.0
license_name: gemma
library_name: transformers
pipeline_tag: image-text-to-text
base_model: llmfan46/G4-MeroMero-26B-A4B-it-uncensored-heretic
tags:
- gemma
- gemma4
- moe
- finetune
- nvfp4
- w4a16
- awq
- mixed-precision
- compressed-tensors
- vllm
language: - en
G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4-W4ATTN-AWQ
Mixed-precision quantization of llmfan46/G4-MeroMero-26B-A4B-it-uncensored-heretic (17.9 GB).
Quantization
| Layer | Format | Group |
|---|---|---|
| Attention (q/k/v/o) | W4A16 (int4) | 128 |
| MLP / MoE experts | NVFP4 (float4) | 16 |
| Embeddings, norms, router | BF16 | — |
- Tooling: llmcompressor
oneshotwithconfig_groupsper-layer dispatch. - Output:
quant_method: "compressed-tensors", two config_groups.
Requirements
Requires 1Cat-vLLM with the feat/gemma4-sm70-native-v3 branch (Gemma-4 model port + SM70 D=512 attention kernels). Untested on standard vLLM.
Usage
from vllm import LLM, SamplingParams
llm = LLM(
"TheUnderscore/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4-W4ATTN-AWQ",
tensor_parallel_size=2,
max_model_len=8192,
)
output = llm.generate("Hello", SamplingParams(temperature=1.0, top_p=0.95, max_tokens=256))
print(output[0].text)
Files
quantize-nvfp4-gemma4.py— quantization script.