license: other
base_model: llmfan46/G4-MeroMero-26B-A4B-it-uncensored-heretic
tags:

• fp8
• llm-compressor
• vllm
• moe

G4-MeroMero-26B FP8 Dynamic Uncensored

This repository contains an FP8 Dynamic quantized version of the original model llmfan46/G4-MeroMero-26B-A4B-it-uncensored-heretic.

Quantization Methodology

The quantization process was conducted using the llm-compressor library from Neural Magic, executing a data-free one-shot dynamic quantization.

To preserve the cognitive logic, output consistency, and potential multimodal projection capabilities of the model, specific critical layers were excluded from quantization and preserved in their native BFloat16 precision:

• Input Embeddings (embed_tokens)
• Output Prediction Head (lm_head)
• Vision Tower / Encoder (vision_tower)
• Vision Projector (multi_modal_projector)
• MoE Routers / Gate layers (gate)

All standard internal Linear projections within the transformer blocks were quantized to FP8 Dynamic (weights are statically quantized to FP8 per-channel, and activations are dynamically quantized to FP8 per-token). This approach yields a 2x reduction in checkpoint size while maintaining near-lossless generation quality (~99.5% retention compared to the BF16 baseline).

Inference Recommendations

For optimal throughput, latency, and memory footprint, we recommend utilizing the vLLM engine, which natively supports dynamic FP8 activation scaling and execution on modern GPU architectures (such as NVIDIA Blackwell B200 / Hopper H100).

Recommended vLLM Launch Configuration

To maximize generation performance and optimize KV cache utilization, launch the vLLM server with the following settings:

python3 -m vllm.entrypoints.openai.api_server \
    --model cloud19/G4-MeroMero-26B-FP8-Dynamic-Uncensored \
    --served-model-name gemma-rp-uncensored \
    --dtype bfloat16 \
    --quantization compressed-tensors \
    --load-format safetensors \
    --kv-cache-dtype fp8 \
    --trust-remote-code

High-Performance Backends

To achieve maximum inference speed, ensure the following environment variables and libraries are active:

• FlashInfer Sampler: Improves sampling speeds significantly.
• SageAttention: Set VLLM_ATTENTION_BACKEND="sageattn" to utilize the SageAttention backend for faster context handling, or fall back to flash_attn on supported hardware.