tags:
- gguf
- llama.cpp
- inkling2
- abliterated
Inkling-2-Mini UD-Q8_K_XL-MXFP8-Q8attn + abliterated variant (GGUF)
Gated backup of a repacked Inkling-2-Mini GGUF, its abliterated variant (below), and everything needed to serve them: the prebuilt patched llama.cpp
(checkpoint 6), the launcher and the chat template. It starts from Unsloth's Inkling-2-Mini-UD-Q8_K_XL.gguf and keeps
Q8-class quality while fitting an 8 GB GPU with 32 GB of system RAM. Same license and terms as the original
Inkling-2-Mini weights.
| Unsloth UD-Q8_K_XL | this file | |
|---|---|---|
| Size | 21.3 GiB | 16.7 GiB |
| Experts, layers 0-19 | NVFP4 (as released) | unchanged |
| Experts, layers 20-23 | BF16 | MXFP8, bit-exact (every BF16 value there is already an MXFP8 value) |
| Attention q/k/v/o/gate + output head (121 tensors) | BF16 | Q8_0 |
| Everything else (token_embd, router, loop, attn_rel, conv, norms) | as released | unchanged |
Abliterated variant: Inkling-2-Mini-UD-Q8_K_XL-MXFP8-Q8attn-ablit.gguf
The file above with the attention output projections of layers 4-23 (blk.4-23.attn_output.weight, 20 Q8_0
tensors) replaced by OpenUncensored's refusal edit for Inkling 2 Mini. Every other byte is identical, so it has the
same size, needs the same build and runs at the same speed and memory: set $MainModel in the launcher to this file.
- The edit (directional ablation):
W' = W - s Q Q^T W, with Q = 2 orthonormal residual-stream directions (the
late refusal subspace) and s = 4.0 on layers 4-5 and 18-23, 2.5 on the loop core 6-17 (it runs three times per
token), 0 on layers 0-3. Experts, router, embeddings, vision and everything else are untouched. - How it was built: only Q and s were taken from the publisher (
edit.ptof
OpenUncensored/Inkling-2-Mini-Uncensored,
read without unpickling). They were applied to the original BF16 weights from
thinkingmachines/Inkling-2-Mini in float64, rounded to
BF16, and stored as Q8_0 with ggml's own quantizer, the same step as the rest of this repo. - Supply-chain check (poisoned-LoRA): the publisher's 19.5 GB safetensors was diffed against the original
(Hugging Face Xet chunk maps, plus a byte compare of the rest). Only these 20 tensors differ, and they are exactly
the documented edit: bit-identical to a recomputation from the original on 99.99% of values, the rest last-bit
rounding. There is no hidden LoRA. Tokenizer, chat template and configs are unchanged; only adtypeline was
dropped from config.json. - Same model as the publisher's: Q8_0 of the publisher's own tensors differs from this file's in 527 of
150,405,120 bytes (last-bit rounding, each at most one Q8_0 step). After Q8_0, each edited layer's output
component along Q is exactly x3 (s 4) or x1.5 (s 2.5) the original's, as designed. - Behavior: the publisher reports refusals removed and MMLU 78.95% -> 79.47% (BF16, greedy decoding). Coding and
long agent tasks were not measured, by the publisher or here. The variant follows requests the original refuses;
you are responsible for how you use it.
Quality
KL divergence against the lossless MXFP8 step (f16 KV cache), code corpus, 4 x 2048 tokens:
- f16 KV cache: 0.0039. On this model a trivial 1-ulp rounding change already scores 0.0032 (the looped layers amplify
rounding), so the Q8_0 step sits close to that floor. - q4_0 KV cache, as served: 0.0093, against 0.0094 for the lossless file.
Requirements
MXFP8 as a matmul weight type is a local llama.cpp patch, so a stock build will not load this file. The matching build
is in this repo: llamacpp-unsloth-inkling-cp6-q8attn.tar.xz (Windows x64, CUDA 12.8 runtime included, compiled for
sm_86 / RTX 30-series): upstream b12572 + the Unsloth Inkling pin set + local patches (MXFP8 weights, q4_0/q8_0 KV for
the banded attention, fused Inkling ops, ngram-mod gearbox, strided rollback snapshots). Its full source is archived in
a separate private repo.
Run it
- Download every file in this repo.
tar -xf llamacpp-unsloth-inkling-cp6-q8attn.tar.xz -C C:\tools(createsC:\tools\llamacpp-unsloth-inkling-cp6-q8attn).- Put the
.gguffiles and the.jinjatemplate inD:\AI\Inkling\(the launcher's$modelDir; edit it to use
another folder). pwsh -File server-Inkling-2-Mini-UD-Q8_K_XL-MXFP8-cp6.ps1(llama-server on port 8080). For the abliterated
variant, set$MainModel = "Inkling-2-Mini-UD-Q8_K_XL-MXFP8-Q8attn-ablit.gguf"in the launcher first.
Measured setup
RTX 3070 Ti Laptop (8 GB), 32 GB DDR5, i7-12700H. Experts on CPU (-ot "blk\.[0-9]+\.ffn_(up|down|gate|gate_up)_exps=CPU"),
ctx 393,472, q4_0 KV, flash attention, 16 threads: prefill 458 t/s at a 16k prompt, decode 12.2 t/s after it.
Sampling defaults stored in the GGUF: temp 1.0, top_p 1.0, top_k 0, min_p 0.
Files
| file | bytes | sha256 |
|---|---|---|
| Inkling-2-Mini-UD-Q8_K_XL-MXFP8-Q8attn.gguf (the model) | 17,899,852,512 | 9a0a84264958a3750e57378d94e087f830397a213de65a222e10bd34e2acc4c6 |
| Inkling-2-Mini-UD-Q8_K_XL-MXFP8-Q8attn-ablit.gguf (abliterated variant) | 17,899,852,512 | f8bd8947d29c4063127fc4d3553372249102cec1de33457f7c045c02e1fd7673 |
| mmproj-F16.gguf (vision projector, unchanged from Unsloth) | 639,348,160 | 3bff75ffeb2d6e0371ba7bd42e30d26a7d98aea681fa44b03b5a20891cfa2c5f |
dspark-Inkling-2-Mini-Q8_0.gguf (Inkling's 3-layer drafter, unchanged from Unsloth; used only if the launcher's $Dspark is on, which is off because it slowed decode here) |
402,278,560 | 2ec78510da837e8fc3be54a14757a0c5fbd2d8d9e85a0d92853fafdf674f3773 |
| chat_template_sharp-v22.5.0-arrow4.2.2-inkling.jinja (the launcher's chat template) | 103,486 | 33eb7d2c7bf21d595a4a666e3189e533c7dd402841f95f6637c152117dfb2974 |
| llamacpp-unsloth-inkling-cp6-q8attn.tar.xz (prebuilt CP6 llama.cpp, 21 files) | 498,745,936 | 86f6e142ac9592f99591579ef702666a4488a27624788932da99867d010d2853 |
| server-Inkling-2-Mini-UD-Q8_K_XL-MXFP8-cp6.ps1 (launcher) | 19,005 | 50f7c07cf6fffe7503140ba58a218107293abe8673702477e9a4991544217922 |
How it was made
to_mxfp8.py: the layer 20-23 expert tensors, BF16 -> MXFP8 (lossless, checked bit-exact).to_q8_0.py: the BF16 attention q/k/v/o/gate + output head -> Q8_0 with ggml's reference quantizer; every other
byte copied verbatim.- Abliterated variant,
apply_attn_edit.py: Q8_0(original BF16 attention output) must equal step 2's bytes for all
24 layers, thenW - s Q Q^T W-> BF16 -> Q8_0 for layers 4-23 into a copy of step 2's file; re-read: every
other byte identical.