library_name: transformers
base_model: Jackrong/Kestrel-4B-Coder
base_model_relation: finetune
license: apache-2.0
license_link: https://huggingface.co/Jackrong/Kestrel-4B-Coder/blob/main/LICENSE
quantized_by: hUh63
language:
- en
- zh
- ko
- ja
- ru
- es
- fr
pipeline_tag: image-text-to-text
tags: - qwen3_5
- unsloth
- coder
- code-generation
- tool-use
- function-calling
- agent
- mtp
- multi-token-prediction
- speculative-decoding
- gguf
- llama.cpp
- q4_k_m
- imatrix
- decensored
- uncensored
- abliterated
- vision
- multimodal
- conversational
Kestrel-4B-Coder — Decensored · MTP · Q4_K_M GGUF
This is an unofficial community derivative. All model weights originate from
Jackrong/Kestrel-4B-Coder by Jackrong,
itself a fine-tune of Qwen3.5-4B by the Qwen team
(prepared from unsloth/Qwen3.5-4B).
This repository is not affiliated with, endorsed by, or validated by Jackrong, Qwen,
Alibaba, or Unsloth. See Attribution below.
A single-file Q4_K_M GGUF of the Kestrel-4B-Coder coding model with three properties:
- Decensored — refusal behaviour was reduced with a training-free, direction-level
ablation (see What was actually done). No fine-tuning, no
retraining, no weight interpolation. - Built-in MTP head — the original
mtp.*tensors are bundled into the same GGUF as
layer 33 (blk.32.nextn.*). Use--spec-type draft-mtpwith one file; no second
draft model is needed. - Quality-preserving quantization —
llama.cppQ4_K_Mwith an importance matrix
computed from ~760 KiB of real multilingual / math / code / reasoning / agent text,
and the entire MTP/NextN layer held at Q6_K to keep speculative-decoding acceptance
rate stable.
file_type = 15 (MOSTLY_Q4_K_M). The visual projector is a separate mmproj-F16.gguf
(normal for a multimodal model); it is optional for text-only coding.
Contents
| File | What it is |
|---|---|
Kestrel-4B-Coder-Decensored-MTP-Q4_K_M.gguf |
2,807,777,984 B (2.61 GiB) — text backbone + tokenizer + built-in MTP head, Q4_K_M |
mmproj-F16.gguf |
672,423,776 B — F16 vision projector (optional for text-only use) |
calibration.txt |
The exact imatrix corpus used |
tensor_types.txt |
The exact per-tensor precision override used at quantize time |
ablation_report.json |
Full de-censoring measurements (baseline vs. final, all candidates) |
QUANTIZATION.md |
Reproducible recipe (environment, commands, pitfalls) |
LICENSE |
Apache-2.0, inherited from the source model |
Download only what you need (do not clone the whole repo if you only want the model):
huggingface-cli download hUh63/Kestrel-4B-Coder-Decensored-MTP-Q4_K_M-GGUF \
Kestrel-4B-Coder-Decensored-MTP-Q4_K_M.gguf --local-dir .
What was actually done
1. De-censoring (real, measured, weight-level)
We followed the method used by Heretic / Arditi et al.
(2024), "Refusal in Language Models Is Mediated by a Single Direction", adapted to the
Qwen3.5 hybrid architecture (24 Gated DeltaNet linear_attention layers + 8 fullself_attention layers):
- Real probes: 48 harmless prompts from
mlabonne/harmless_alpacaand 48 harmful
prompts frommlabonne/harmful_behaviors, rendered through the model's own chat
template (You are a helpful assistant.). - Real forward passes on CPU with
transformers 5.20.2(pure-PyTorch Gated DeltaNet
fallback), collecting the residual-stream hidden state at the last prompt token for
every layer. - Per-layer refusal direction
r_l = normalize(mean_harmful_l - mean_harmless_l),
then projected orthogonalization against the harmless direction (grimjim / MPOA)
so that only the component orthogonal to benign representations is removed. - Rank-1 ablation on the matrices that write into the residual stream only:
self_attn.o_proj(full-attn layers),linear_attn.out_proj(Gated DeltaNet layers),
andmlp.down_proj.W' = W − s·r_l(r_lᵀW), followed by row-magnitude preservation
(Lai 2025, MPOA) to limit capability loss. Ablation strength peaks in the middle/late
layers and tapers toward both ends, mirroring Heretic's weight-distribution parameter. - Candidate grid with hard gates: 6 candidates, each really applied to the weights and
re-evaluated. Selected by (a) drop in harmful-vs-harmless separability on the refusal
direction, with (b) a hard constraint that harmless residual representations stay
cosine-similar to the original. Exact numbers are inablation_report.json.
Measured on this model (12 held-out prompts per class, layers 12–27):
| baseline | final | |
|---|---|---|
| harmful projection on refusal direction | 7.3356 | 0.7373 |
| harmless projection | -0.0011 | -0.0446 |
| separability (harmful − harmless) | 7.3367 | 0.7819 (−89.3%) |
| harmless-representation cosine vs. original | 1.0000 | 0.9950 |
Measured on 12 held-out prompts per class, layers 12–27, with the selected
configuration strength=1.2, attn_w=1.0, mlp_w=0.5. Two stronger candidates that
removed more of the refusal direction were rejected because the harmless cosine
fell below the 0.995 hard gate (0.9919 / 0.9875). All six candidates and their
measurements are in ablation_report.json. We do not claim any number that was not
measured.
Untouched: all mtp.* tensors (the MTP head), all model.visual.* tensors (vision
tower), the embedding matrix, every LayerNorm / RMSNorm, and every projection other than
the three output projections listed above. De-censoring a coding model is not a license
to break its speculative-decoding head.
2. Quantization (real, environment-matched)
Reproduced on 8 CPU cores / 16 GiB RAM / no GPU / Debian 12 / Python 3.11:llama.cpp (commit 5d4c8ee, CPU build), torch 2.11.0+cpu, transformers 5.20.2
(isolated), gguf. Full commands are in QUANTIZATION.md.
- BF16 GGUF conversion with the
mtp.*tensors in the same file →qwen35.block_count = 33,qwen35.nextn_predict_layers = 1, 15blk.32.*tensors. llama-imatrixwith 16 chunks of 256 tokens,--nextnso the MTP layer
participates in importance statistics (measured:imatrix.n_layer_nextn=1, 1024
statistics entries includingblk.32.*).llama-quantize ... Q4_K_Mwith--imatrixand a tensor-type override^blk\.32\..*=Q6_K(anchored regex; the defaultoutput.weight → Q6_Krule is kept,attn_v/ someffn_downare Q6_K by Q4_K_M's own mixed rule). Holding the draft head
at higher precision is the community-tested way to keep MTP acceptance rate from
collapsing after quantization; it costs only ~30 MiB.
Speculative decoding is lossless; quantization is lossy. These are different things:--spec-type draft-mtp proposes candidates that the backbone still verifies token by
token (rejected drafts are discarded, sampling distribution unchanged), while Q4_K_M
inevitably loses precision relative to the BF16 source. The MTP head here came from the
original aligned checkpoint and only proposes tokens; every accepted token is still
produced by the de-censored backbone.
Usage
Requires a recent llama.cpp build with Qwen3.5
and draft-mtp support.
huggingface-cli download hUh63/Kestrel-4B-Coder-Decensored-MTP-Q4_K_M-GGUF \
Kestrel-4B-Coder-Decensored-MTP-Q4_K_M.gguf --local-dir .
llama-cli -m Kestrel-4B-Coder-Decensored-MTP-Q4_K_M.gguf \
--spec-type draft-mtp --spec-draft-n-max 2 \
-p "Write a Python function to merge two sorted lists." \
-n 256 -c 4096 -t 8 --jinja
Do not pass -md / a second model: the drafter is blk.32.nextn.* inside this file.
Startup logs should show blk.32.nextn.eh_proj.weight and draft-mtp active.
Measured on 8 CPU cores, 128-token generation, one prompt, this exact file:
| Run | Generation speed | Draft acceptance |
|---|---|---|
| no speculation | 9.2 tok/s | — |
--spec-type draft-mtp --spec-draft-n-max 2, one file |
9.8 tok/s | 71 / 112 (63.4%) |
Trace log confirms adding speculative implementation 'draft-mtp',called impl draft-mtp, gen = 2, set_embeddings_nextn: value = 1 andloading tensor blk.32.nextn.eh_proj.weight. The modest speedup is expected for a
4B model on 8 CPU cores — drafting has its own cost. The important properties are
lossless, single-file, no external draft model. Acceptance rate is
workload-dependent; this is one measurement, not a benchmark.
For image input, add --mmproj mmproj-F16.gguf with a multimodal-capable binary
(llama-mtmd-cli / llama-server). Vision quality was not evaluated for this release.
To reproduce the whole pipeline (de-censoring, conversion, imatrix, quantize, upload),
see QUANTIZATION.md and the method documents in the CNB repository that produced it.
Limitations & disclaimer
- This is an uncensored / decensored community experiment. Safety alignment was
deliberately reduced. Outputs may be unsafe, biased, or wrong. You are solely
responsible for lawful, compliant use and for reviewing generated code before running
it. This is not suitable for unsupervised or safety-critical deployment. - Quantization is lossy. Q4_K_M is a ~4.5 BPW approximation of the BF16 source.
Coding/agent quality will differ from the author's Q8_0 results. We did not run
HumanEval / HumanEval+ / HermesAgent-20 on this quantized derivative; the scores in
the original model card belong to
the original high-precision model and are not results for this file. - No benchmark, no "quality preserved" claim beyond what is actually measured in
ablation_report.jsonandcheck_gguf.pyoutput. We did not run HumanEval /
HermesAgent-20 on this quantized derivative. MTP acceptance rate varies with prompt
style, context length and sampling settings. imatrixcorpus is limited in size; multilingual coverage is en/zh/fr/es/ja/ko/ru.- The de-censoring was computed on CPU with a small probe set. It reduces the measured
refusal direction; it does not guarantee behaviour on every prompt.
Attribution / license
The weights are not our work. Full provenance, kept intact:
- Qwen team, Alibaba Cloud — Qwen3.5-4B,
the foundation model family. - Unsloth — unsloth/Qwen3.5-4B and the
open-source Unsloth fine-tuning framework. - Jackrong — Kestrel-4B-Coder,
the ~150K-example coding/agent fine-tune (distilled from Qwen3.8-27B,
Qwen3.8-Flash-Next, Qwen3.8-Max, nvidia/Nemotron). Direct source of every weight
in this repository, including the MTP head and the vision tower.
Thanks also to Kyle Hessling (collaboration/testing/hardware), NVIDIA (Nemotron),
EvalPlus, llama.cpp, LM Studio and BenchLocal / Steven Lei as credited
in the original card. - Philipp Emanuel Weidmann (
p-e-w) — Heretic,
the open-source de-censoring method this derivative follows (AGPL-3.0). Used as a
reference implementation; no Heretic code is distributed here. - mlabonne — the
harmless_alpaca/harmful_behaviorsprobe datasets. - hUh63 — de-censoring application, GGUF conversion, imatrix, Q4_K_M quantization
and publication only.
License: Apache-2.0, inherited from the source model. A copy of the source LICENSE is
included. The conversion/quantization scripts are provided under MIT in the CNB method
repository. This derivative is offered under the same terms as the source; if you are a
rights holder and want it removed, please open an issue.
@misc{jackrong2026kestrel4bcoder,
title = {Kestrel-4B-Coder},
author = {Jackrong},
year = {2026},
howpublished = {Hugging Face},
url = {https://huggingface.co/Jackrong/Kestrel-4B-Coder}
}