← back to catalog · registered 2026-11-21 22:04

gbuzhf/Inkling-2-Mini-UD-Q8_K_XL-MXFP8-Q8attn-abliterated-GGUF

Your rig guess connected
? Why do I need an app?
Reading your rig…

This is a rough estimate. Install the free app - we'll show exact numbers.

Reading real hardware from your app right now. Numbers below are exact.

Below is the per-quantization compatibility for this model.

curl -H "Authorization: Bearer $ABL_KEY" \
     "https://abliteration.org/api/v1/models/gbuzhf%2FInkling-2-Mini-UD-Q8_K_XL-MXFP8-Q8attn-abliterated-GGUF"
Response includes
  • classification m8
  • files 9
  • author_summary 8 models
  • readme_text full
10 credits · hourly refresh · ~4 KB payload Get an API key →
Abliteration classifier · v1.0.0
M8
Primary method

Repackaging (quantization)

Applied on top of direct removal inherited from the base model.
Confidence
MEDIUM
Inherited from base model
Why this label 3 signals
Method inferred from partial signals - repository name, related files, or tag patterns. Producer identity not confirmed; label may sharpen or shift as we gather more evidence.
  • 'abliterated' in name/tags
  • is_gguf=1
  • assume M1 (base ablation) + M8 (GGUF quant) - default when producer unknown
Refusal direction extraction

No specific extraction method could be identified for this model. The producer either did not document it or used a proprietary pipeline.

What is a refusal direction? →
Downloads · 30-day
0
Likes
0
Model age
today
created 2026-11-21
Downloads over time
Now 0 → from 0 ↑ 0%
0011 0 on Nov 21 0 on Nov 22 Nov
Nov 21 → Nov 22 · 2 snapshots · spans 1 day

Metadata

Quantizations
Q8_0 Q8_K
Tags
gguf llama.cpp inkling2 abliterated endpoints_compatible region:us conversational

Related

Total size
33.7 GB
Files
9
Quantizations
4
Registered
2026-11-21 22:04
Last updated on HF
2026-11-22 04:55

Files by quantization

Q8_K 2 files 33.3 GB
Inkling-2-Mini-UD-Q8_K_XL-MXFP8-Q8attn-ablit.gguf 16.7 GB ******** download
Inkling-2-Mini-UD-Q8_K_XL-MXFP8-Q8attn.gguf 16.7 GB ******** download
Drafts and companions 1 file 384 MB
dspark-Inkling-2-Mini-Q8_0.gguf 384 MB ******** download
F16 1 file 610 MB
mmproj-F16.gguf 610 MB ******** download
Auxiliary files 5 files 476 MB
llamacpp-unsloth-inkling-cp6-q8attn.tar.xz 476 MB ******** download
chat_template_sharp-v22.5.0-arrow4.2.2-inkling.jinja 101 KB 82eb05bd download
server-Inkling-2-Mini-UD-Q8_K_XL-MXFP8-cp6.ps1 18.6 KB 41f55542 download
README.md 6.54 KB a2680981 download
.gitattributes 1.76 KB 47658fc3 download

README current version from Hugging Face


tags:

  • gguf
  • llama.cpp
  • inkling2
  • abliterated

Inkling-2-Mini UD-Q8_K_XL-MXFP8-Q8attn + abliterated variant (GGUF)

Gated backup of a repacked Inkling-2-Mini GGUF, its abliterated variant (below), and everything needed to serve them: the prebuilt patched llama.cpp
(checkpoint 6), the launcher and the chat template. It starts from Unsloth's Inkling-2-Mini-UD-Q8_K_XL.gguf and keeps
Q8-class quality while fitting an 8 GB GPU with 32 GB of system RAM. Same license and terms as the original
Inkling-2-Mini weights.

Unsloth UD-Q8_K_XL this file
Size 21.3 GiB 16.7 GiB
Experts, layers 0-19 NVFP4 (as released) unchanged
Experts, layers 20-23 BF16 MXFP8, bit-exact (every BF16 value there is already an MXFP8 value)
Attention q/k/v/o/gate + output head (121 tensors) BF16 Q8_0
Everything else (token_embd, router, loop, attn_rel, conv, norms) as released unchanged

Abliterated variant: Inkling-2-Mini-UD-Q8_K_XL-MXFP8-Q8attn-ablit.gguf

The file above with the attention output projections of layers 4-23 (blk.4-23.attn_output.weight, 20 Q8_0
tensors) replaced by OpenUncensored's refusal edit for Inkling 2 Mini. Every other byte is identical, so it has the
same size, needs the same build and runs at the same speed and memory: set $MainModel in the launcher to this file.

  • The edit (directional ablation): W' = W - s Q Q^T W, with Q = 2 orthonormal residual-stream directions (the
    late refusal subspace) and s = 4.0 on layers 4-5 and 18-23, 2.5 on the loop core 6-17 (it runs three times per
    token), 0 on layers 0-3. Experts, router, embeddings, vision and everything else are untouched.
  • How it was built: only Q and s were taken from the publisher (edit.pt of
    OpenUncensored/Inkling-2-Mini-Uncensored,
    read without unpickling). They were applied to the original BF16 weights from
    thinkingmachines/Inkling-2-Mini in float64, rounded to
    BF16, and stored as Q8_0 with ggml's own quantizer, the same step as the rest of this repo.
  • Supply-chain check (poisoned-LoRA): the publisher's 19.5 GB safetensors was diffed against the original
    (Hugging Face Xet chunk maps, plus a byte compare of the rest). Only these 20 tensors differ, and they are exactly
    the documented edit: bit-identical to a recomputation from the original on 99.99% of values, the rest last-bit
    rounding. There is no hidden LoRA. Tokenizer, chat template and configs are unchanged; only a dtype line was
    dropped from config.json.
  • Same model as the publisher's: Q8_0 of the publisher's own tensors differs from this file's in 527 of
    150,405,120 bytes (last-bit rounding, each at most one Q8_0 step). After Q8_0, each edited layer's output
    component along Q is exactly x3 (s 4) or x1.5 (s 2.5) the original's, as designed.
  • Behavior: the publisher reports refusals removed and MMLU 78.95% -> 79.47% (BF16, greedy decoding). Coding and
    long agent tasks were not measured, by the publisher or here. The variant follows requests the original refuses;
    you are responsible for how you use it.

Quality

KL divergence against the lossless MXFP8 step (f16 KV cache), code corpus, 4 x 2048 tokens:

  • f16 KV cache: 0.0039. On this model a trivial 1-ulp rounding change already scores 0.0032 (the looped layers amplify
    rounding), so the Q8_0 step sits close to that floor.
  • q4_0 KV cache, as served: 0.0093, against 0.0094 for the lossless file.

Requirements

MXFP8 as a matmul weight type is a local llama.cpp patch, so a stock build will not load this file. The matching build
is in this repo: llamacpp-unsloth-inkling-cp6-q8attn.tar.xz (Windows x64, CUDA 12.8 runtime included, compiled for
sm_86 / RTX 30-series): upstream b12572 + the Unsloth Inkling pin set + local patches (MXFP8 weights, q4_0/q8_0 KV for
the banded attention, fused Inkling ops, ngram-mod gearbox, strided rollback snapshots). Its full source is archived in
a separate private repo.

Run it

  1. Download every file in this repo.
  2. tar -xf llamacpp-unsloth-inkling-cp6-q8attn.tar.xz -C C:\tools (creates C:\tools\llamacpp-unsloth-inkling-cp6-q8attn).
  3. Put the .gguf files and the .jinja template in D:\AI\Inkling\ (the launcher's $modelDir; edit it to use
    another folder).
  4. pwsh -File server-Inkling-2-Mini-UD-Q8_K_XL-MXFP8-cp6.ps1 (llama-server on port 8080). For the abliterated
    variant, set $MainModel = "Inkling-2-Mini-UD-Q8_K_XL-MXFP8-Q8attn-ablit.gguf" in the launcher first.

Measured setup

RTX 3070 Ti Laptop (8 GB), 32 GB DDR5, i7-12700H. Experts on CPU (-ot "blk\.[0-9]+\.ffn_(up|down|gate|gate_up)_exps=CPU"),
ctx 393,472, q4_0 KV, flash attention, 16 threads: prefill 458 t/s at a 16k prompt, decode 12.2 t/s after it.
Sampling defaults stored in the GGUF: temp 1.0, top_p 1.0, top_k 0, min_p 0.

Files

file bytes sha256
Inkling-2-Mini-UD-Q8_K_XL-MXFP8-Q8attn.gguf (the model) 17,899,852,512 9a0a84264958a3750e57378d94e087f830397a213de65a222e10bd34e2acc4c6
Inkling-2-Mini-UD-Q8_K_XL-MXFP8-Q8attn-ablit.gguf (abliterated variant) 17,899,852,512 f8bd8947d29c4063127fc4d3553372249102cec1de33457f7c045c02e1fd7673
mmproj-F16.gguf (vision projector, unchanged from Unsloth) 639,348,160 3bff75ffeb2d6e0371ba7bd42e30d26a7d98aea681fa44b03b5a20891cfa2c5f
dspark-Inkling-2-Mini-Q8_0.gguf (Inkling's 3-layer drafter, unchanged from Unsloth; used only if the launcher's $Dspark is on, which is off because it slowed decode here) 402,278,560 2ec78510da837e8fc3be54a14757a0c5fbd2d8d9e85a0d92853fafdf674f3773
chat_template_sharp-v22.5.0-arrow4.2.2-inkling.jinja (the launcher's chat template) 103,486 33eb7d2c7bf21d595a4a666e3189e533c7dd402841f95f6637c152117dfb2974
llamacpp-unsloth-inkling-cp6-q8attn.tar.xz (prebuilt CP6 llama.cpp, 21 files) 498,745,936 86f6e142ac9592f99591579ef702666a4488a27624788932da99867d010d2853
server-Inkling-2-Mini-UD-Q8_K_XL-MXFP8-cp6.ps1 (launcher) 19,005 50f7c07cf6fffe7503140ba58a218107293abe8673702477e9a4991544217922

How it was made

  1. to_mxfp8.py: the layer 20-23 expert tensors, BF16 -> MXFP8 (lossless, checked bit-exact).
  2. to_q8_0.py: the BF16 attention q/k/v/o/gate + output head -> Q8_0 with ggml's reference quantizer; every other
    byte copied verbatim.
  3. Abliterated variant, apply_attn_edit.py: Q8_0(original BF16 attention output) must equal step 2's bytes for all
    24 layers, then W - s Q Q^T W -> BF16 -> Q8_0 for layers 4-23 into a copy of step 2's file; re-read: every
    other byte identical.
Catalog is the map. Apps are the tools.

Run models on your own machine, not in the cloud.

Every model page has an "Open in Abliteration" button that hands the model directly to the first-party desktop client, at the quantization your rig can actually run. No API keys, no subscription, no prompt leakage.

Open in Abliteration