license: apache-2.0
base_model: llmfan46/gemma-4-31B-it-uncensored-heretic
tags:
- fp8
- quantized
- compressed-tensors
- llmcompressor
- gemma4
gemma-4-31B-it-uncensored-heretic-FP8
FP8 block-quantized version of llmfan46/gemma-4-31B-it-uncensored-heretic.
Quantization Details
- Method: FP8_BLOCK (weight_block_size=[128,128], activation_scheme=dynamic)
- Tool: llmcompressor
model_free_ptq - Format: compressed-tensors (vLLM native)
- Ignored layers: vision tower, lm_head, embedding layers
- Model size: ~31.5 GB (vs ~62 GB BF16)
Evaluation Results
Compared against RedHatAI/gemma-4-31B-it-FP8-block using identical evaluation settings (lm_eval 0.4.11, vLLM, 2 seeds averaged).
| Benchmark | This Model | RedHatAI FP8 | Delta |
|---|---|---|---|
| GSM8k-Platinum (5-shot) | 97.68% | 97.82% | -0.14 |
| MMLU-CoT (5-shot) | 89.97% | 90.70% | -0.73 |
| MMLU-Pro (5-shot) | 84.52% | 84.92% | -0.40 |
| IFEval inst_strict | 93.41% | 93.84% | -0.43 |
| IFEval prompt_strict | 90.76% | 91.31% | -0.55 |
Generation parameters: temperature=1.0, top_p=0.95, top_k=64, max_gen_toks=8192
Usage
from vllm import LLM
model = LLM("kakrotto/gemma-4-31B-it-uncensored-heretic-FP8")
Disclaimer
This is an uncensored model. The quantizer (kakrotto) is not responsible for the model's outputs or any misuse. This FP8 quantization preserves the original model's behavior. Please use responsibly.
Attribution
- Source model: llmfan46/gemma-4-31B-it-uncensored-heretic
- Quantization recipe reference: RedHatAI/gemma-4-31B-it-FP8-block