license: apache-2.0
base_model: llmfan46/gemma-4-31B-it-uncensored-heretic
tags:
- fp8
- quantized
- compressed-tensors
- llmcompressor
- gemma4
gemma-4-31B-it-uncensored-heretic-FP8
FP8 block-quantized version of llmfan46/gemma-4-31B-it-uncensored-heretic.
Quantization Details
- Method: FP8_BLOCK (weight_block_size=[128,128], activation_scheme=dynamic)
- Tool: llmcompressor
model_free_ptq - Format: compressed-tensors (vLLM native)
- Ignored layers: vision tower, lm_head, embedding layers
- Model size: ~31.5 GB (vs ~62 GB BF16)
Evaluation Results
Compared against RedHatAI/gemma-4-31B-it-FP8-block using identical evaluation settings (lm_eval 0.4.11, vLLM, 2 seeds averaged).
| Benchmark | This Model | RedHatAI FP8 | Delta |
|---|---|---|---|
| GSM8k-Platinum (5-shot) | 97.68% | 97.82% | -0.14 |
| MMLU-CoT (5-shot) | 89.97% | 90.70% | -0.73 |
| MMLU-Pro (5-shot) | 84.52% | 84.92% | -0.40 |
| IFEval inst_strict | 93.41% | 93.84% | -0.43 |
| IFEval prompt_strict | 90.76% | 91.31% | -0.55 |
Generation parameters: temperature=1.0, top_p=0.95, top_k=64, max_gen_toks=8192
Usage
from vllm import LLM
model = LLM("m672/gemma-4-31b-it-uncensored-fp8")
Disclaimer
This is an uncensored model. The quantizer (kakrotto) is not responsible for the model's outputs or any misuse. This FP8 quantization preserves the original model's behavior. Please use responsibly.
Attribution
- Source model: llmfan46/gemma-4-31B-it-uncensored-heretic
- Quantization recipe reference: RedHatAI/gemma-4-31B-it-FP8-block
Copy provenance and thinking default
This repository copies the inference files from kakrotto/gemma-4-31B-it-uncensored-heretic-FP8 at commit ac337a78d32b874e4ac10aa7f989b1bddd7f145c.
The model weights, quantization, tokenizer, and configuration files are unchanged. The chat template defaults enable_thinking to true when the argument is omitted. Explicitly setting enable_thinking=false disables thinking. Pass this option as a chat-template argument in your inference client or server.