library_name: mlx
base_model: huihui-ai/Huihui-gemma-4-26B-A4B-it-abliterated
tags:
- mlx
- mlx-vlm
- moe
- multimodal
- vision
- gemma
- abliterated
- uncensored
- basequant-xl
pipeline_tag: image-text-to-text
leonsarmiento/Huihui-gemma-4-26B-A4B-it-abliterated-3bit-XL-mlx
This model was converted to MLX format from huihui-ai/Huihui-gemma-4-26B-A4B-it-abliterated using BaseQuant_XL 3-bit mixed quantization — the compact XL variant built for RAM-constrained Macs. The vision encoder is preserved (full multimodal). The unquantized BF16 base was used as the source.
This is an abliterated (uncensored) variant of Gemma-4-26B-A4B: the refusal mechanism has been removed via directional ablation, otherwise identical to the vanilla model — 25.2B-parameter MoE with 128 experts per layer (avg. 3.8B active per token) and a vision encoder.
BaseQuant_XL keeps the most routing-critical layers in full bf16 precision — the MoE router.proj (gate) and the mlp (shared/dense expert that processes all tokens) — while applying 8-bit quantization to embeddings and attention. Only the routed experts (experts.switch_glu), the bulk of the parameters, are pushed down to 3-bit. This is the same layer assignment as the 6-bit XL abliterated and the vanilla 3-bit XL variants — only the routed-expert bit depth differs.
Note: tie_word_embeddings=True means there is no separate lm_head (tied to embed_tokens).
Use with mlx
pip install -U mlx-vlm
python -m mlx_vlm.generate --model leonsarmiento/Huihui-gemma-4-26B-A4B-it-abliterated-3bit-XL-mlx --max-tokens 256 --temperature 1.0 --top-p 1.0 --prompt "Hello"
BaseQuant_XL Quantization Strategy
| Bit Depth | Layers | Rationale |
|---|---|---|
| bf16 (unquantized) | router.proj (gate), mlp (shared/dense expert), vision tower |
Routing decisions and the dense expert that processes every token — errors here cascade through all tokens |
| 8-bit | embed_tokens, self_attn |
Near-lossless for the attention path — the sensitive non-expert weights stay protected |
| 3-bit | experts.switch_glu (routed experts) |
The bulk of parameters (only ~4B of 26B active per token) — 3-bit keeps the footprint small |
Quantization Details
| Layer | Bits | Group Size |
|---|---|---|
router.proj (gate) |
bf16 | — |
mlp (shared/dense expert) |
bf16 | — |
| Vision tower (full) | bf16 | — |
embed_tokens |
8 | 64 |
self_attn |
8 | 64 |
experts.switch_glu (routed experts) |
3 | 64 |
| Default fallback | 8 | 64 |
- Quantization type: BaseQuant_XL mixed (multimodal, vision preserved)
- Bits per weight: 4.404
- Total size: ~13 GB (3 shards)
- Group size: 64
- Method: Custom
quant_predicateviamlx_vlm - Note:
tie_word_embeddings=True— no separate lm_head
Thinking Mode (LM Studio setup)
Thinking is enabled by default in this repo — the flags enable_thinking = true and preserve_thinking = true are baked into the chat template (both chat_template.jinja and tokenizer_config.json), so servers that don't pass template kwargs (LM Studio, oMLX) get thinking out of the box. Passing an explicit enable_thinking=false from a server that supports template kwargs still overrides the default.
The chat template is the canonical Google upstream version (pulled from google/gemma-4-26B-A4B-it at build time — not the fork's copy).
For LM Studio to render the reasoning trace correctly, set the model's Reasoning Parser (in the model settings) to custom strings:
| Setting | Value |
|---|---|
| Reasoning parser start string | <|channel>thought |
| Reasoning parser end string | <channel|> |
The model opens each reasoning block with <|channel>thought and closes it with <channel|> before emitting the final answer — without these parser strings, LM Studio will show the raw channel markers inline with the response.
Recommended Inference Parameters
| Parameter | Value |
|---|---|
temperature |
1.0 |
top_p |
1.0 |
top_k |
64 |
min_p |
0.01 |
repeat_penalty |
1.1 |