base_model: tpls/gemma-4-12B-coder-fable5-composer2.5-v1-abliterated-sft-v5
base_model_relation: quantized
library_name: gguf
pipeline_tag: text-generation
license: gemma
quantized_by: triplezrobotics
tags:

• gguf
• llama.cpp
• gemma4
• abliterated
• code

gemma-4-12B-coder-fable5-composer2.5-v1-abliterated-sft-v5-GGUF

GGUF quantizations of tpls/gemma-4-12B-coder-fable5-composer2.5-v1-abliterated-sft-v5.

Quantized with llama.cpp using an importance matrix (imatrix) calibrated on
code + tool-call markup, with token-embedding and output tensors kept at f16
to protect the tool-call token logits, plus a gemma-4 EOG patch (eot=106) to
bound the <|turn> runaway.

Files

Use it

# llama.cpp (server, with native tool parsing)
llama-server -hf tpls/gemma-4-12B-coder-fable5-composer2.5-v1-abliterated-sft-v5-GGUF:Q4_K_M --jinja --ctx-size 16384

# Ollama
ollama run hf.co/tpls/gemma-4-12B-coder-fable5-composer2.5-v1-abliterated-sft-v5-GGUF:Q4_K_M

⚠️ Tool-calling note (abliterated weights)

The base model and fine-tune emit canonical tool_calls that llama.cpp --jinja
parses natively. Abliteration degrades the canonical tool token: this model
tends to leak its calls as text markup (<|tool>NAME{...}) instead of the
canonical <|tool_call> token, so the native parser may not fire.

Two ways to get structured tool_calls back:

1. Runtime shim (works now). Put a tolerant recovery shim in front via LiteLLM:

   # litellm config
   model_list:
     - model_name: gemma-coder
       litellm_params:
         model: openai/gemma-coder
         api_base: http://127.0.0.1:8096/v1
         api_key: sk-none
   litellm_settings:
     callbacks: ["gemma_tool_shim.shim"]   # recovers tool_calls from leaked markup
     drop_params: true
   

   The shim is published with the source at the repo that produced this model.

2. Native (no shim). Use a control-vector / activation-steered variant
   instead of weight-ablation — steering uncensors at inference without editing
   the weights, so the canonical tool token survives and bare --jinja works.