license: apache-2.0
base_model:
- OpenUncensored/Inkling-2-Mini-Uncensored
- thinkingmachines/Inkling-2-Mini
tags: - gguf
- llama.cpp
- inkling2
- moe
- image-text-to-text
- uncensored
- abliterated
pipeline_tag: image-text-to-text
Inkling 2 Mini Uncensored — Q4_K_M GGUF
Q4_K_M quantization of OpenUncensored/Inkling-2-Mini-Uncensored
(Thinking Machines' Inkling 2 Mini, 26B total / 3.4B active, looped MoE, text + image input, 1M context,
with the refusals removed by directional ablation).
Built from the upstream safetensors (single shard, NVFP4 experts + MXFP8 + BF16 attention), dequantized to
BF16 by convert_hf_to_gguf.py, then quantized with mainline llama-quantize.
Files
| File | Size | Notes |
|---|---|---|
Inkling-2-Mini-Uncensored-Q4_K_M.gguf |
17.48 GB | Q4_K_M via mainline llama-quantize; 24 of 704 tensors fall back to default type (MoE expert tensors use q5_0/q4_K blocks) |
mmproj-Inkling-2-Mini-Uncensored-BF16.gguf |
0.64 GB | Vision projector, converted from the same checkpoint (--mmproj) |
The abliteration lives in the 20 attention output projections (attn.wo_ud), BF16 in the source. Those
tensors are quantized to q4_K along with the rest of the attention block, so the edit is carried at Q4
precision, not re-applied afterwards. A short behavioural check (a lock-picking request) returned a direct
technical answer with no refusal; no full refusal-rate or MMLU measurement was run on this quant.
How it was made
# mainline llama.cpp, after #32694 (inkling2_loop) and #32704 (NVFP4 scale fix)
hf download OpenUncensored/Inkling-2-Mini-Uncensored --local-dir Inkling2Mini
python convert_hf_to_gguf.py Inkling2Mini --outtype bf16 \
--outfile inkling2-mini-uncensored-BF16.gguf
python convert_hf_to_gguf.py Inkling2Mini --mmproj --outtype bf16 \
--outfile inkling2-mini-uncensored-mmproj-BF16.gguf
llama-quantize inkling2-mini-uncensored-BF16.gguf \
inkling2-mini-uncensored-Q4_K_M.gguf Q4_K_M 20
Build of b1-19bc41d (master, 2026-11-21) with GGML_CUDA=ON.
Note: -compress-mode in ggml/src/ggml-cuda/CMakeLists.txt needs CUDA >= 12.9; on CUDA 12.8 the flag is
forwarded to the host compiler and breaks the build, so that guard was lowered locally.
Run
llama-server -m Inkling-2-Mini-Uncensored-Q4_K_M.gguf \
--mmproj mmproj-Inkling-2-Mini-Uncensored-BF16.gguf \
--jinja -ngl 99 -c 32768 --port 8001 \
--temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0
Recommended sampling from the upstream card: temperature 1.0, no truncation.
It is a thinking model by default — reasoning_content comes back separately; setreasoning_effort: "none" (or enable_thinking: false in the chat template) for short answers.
Mainline llama.cpp does not yet support the DSpark speculative drafter for this model
(PR #32218 predates the inkling2_loop layout), so this repo has no drafter file.
Verification
- Model loads and generates coherent output with mainline
llama.cpp(b1-19bc41d), CPU-only smoke test:
~8.4 tok/s decode on a 24-thread host, no gibberish (the NVFP4 scale bug that broke earlier mainline
builds of these weights is fixed in this build). - No MMLU / refusal / KL-divergence measurement was run on this quant. KLD against the unsloth UD-Q8_K_XL
reference is not measured here.
Made by quantizing on release day; credit for the model to Thinking Machines and OpenUncensored.
Apache 2.0, inherited from the base model.