license: apache-2.0
base_model: AtomicChat/Inkling-2-Mini-Uncensored-GGUF
base_model_relation: quantized
pipeline_tag: text-generation
tags:
- gguf
- llama.cpp
- inkling2
- moe
- 2-bit
- uncensored
- rtx-3090
Inkling 2 Mini Uncensored, 2-bit experts for one RTX 3090
A speed-oriented GGUF of AtomicChat's uncensored build of
thinkingmachines/Inkling-2-Mini. It uses the same
2-bit expert layout as our regular build and
decodes at about 200 tokens per second on a single RTX 3090 with our llama.cpp fork.
What "uncensored" means here: this is AtomicChat's edit, not ours. They describe it as a rank-1
LoRA on the 49 places that write into the residual stream (the attention output and routed experts of all
24 layers, plus the token embedding). In each layer it removes that layer's own refusal direction. See their card,
AtomicChat/Inkling-2-Mini-Uncensored-GGUF,
for what the edit does and how they evaluated it. The model will follow requests that the original declines.
You are responsible for how you use it.
Write-up: Looping on a 3090 (link coming soon) · Code:
csaben/llama.cpp, branch inkling-3090
File
| File | Size | SHA-256 |
|---|---|---|
Inkling-2-Mini-Uncensored-Q2K-Q2_0.gguf |
9.25 GiB | 76aaaac94aa25fa517a34e16b2213296f3c0f209d70c32e9949989625ccc0c5b |
Text only. No vision projector is included.
How it was made
The source is AtomicChat's baked Inkling-2-Mini-Uncensored-AD-Q4_K.gguf, at repository revision40886ab636cabb4decf4998224f790b462ab048e. It was requantized with the same per-tensor recipe as the regular
build (all 704 tensor types match):
| Tensors | Type |
|---|---|
Routed experts, layers 0–19: ffn_gate_up_exps |
Q2_K |
Routed experts, layers 0–19: ffn_down_exps |
Q2_0, calibrated |
| Routed experts, layers 20–23 | IQ4_NL |
| Attention projections and output head | IQ4_NL |
| Token embeddings | Q8_0 |
| Norms, router, loop controllers, biases | F32 |
Because the source was already a 4-bit checkpoint, this file has passed through two rounds of quantization. The
regular build was quantized directly from Q8_0. The calibrated Q2_0 encoder is in commit704a7da. The on-disk layout is standard Q2_0.
Measured on one RTX 3090
These were measured with the inkling-3090 fork and the flags below. Perplexity uses the fork's fixed text
(perf/ppl.txt) and measures language-modeling quality, not refusal behavior.
| Model | Decode tok/s, empty context | at 1K context | Prefill PPL (8×2048) | Decode PPL (2×1024, ubatch 1) |
|---|---|---|---|---|
| Original Inkling 2 Mini, ggml-org IQ4_NL (unfused reference) | — | — | 2.9401 | 4.0660 |
| Regular 2-bit build | 208.7 | 193.2 | 3.0530 | 4.1945 |
| This model | 207.9 | 193.2 | 3.0571 | 4.1918 |
Speed drops as the context fills, because every one of the 48 layer passes rereads the attention cache. Expect
about 120–140 tok/s at the 16K–48K contexts coding agents use. The regular build's card has the full table.
Run it
git clone -b inkling-3090 https://github.com/csaben/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF
cmake --build build -j --target llama-server
export LLAMA_INKLING_CTRL_F16=1 IK_BATCH_MESSAGES=1 IK_SEGMENTED_IQ4=1 IK_SEGMENTED_Q6=1 IK_MIX_SPECIALIZE=1 IK_CTRL_APPLY8=1
build/bin/llama-server -m Inkling-2-Mini-Uncensored-Q2K-Q2_0.gguf -ngl 99 -fa on -ctk q8_0 -ctv q8_0 \
-c 65536 -np 1 --backend-sampling --port 8031
Stock llama.cpp with Inkling 2 Mini support
(#32694) also loads the file, but much more slowly.
Limitations
- Two quantization rounds, with perplexity about 4% above the 4-bit original. Check it on your own tasks.
- Refusal behavior comes from AtomicChat's edit. We did not evaluate it.
- Text only. The speed depends on the fork's Ampere-tuned (sm_86) fused kernels.
License and credits
Apache-2.0, matching the base model and AtomicChat's build. This is a quantized derivative. We made no changes
beyond quantization.
- Base model: thinkingmachines/Inkling-2-Mini, Thinking Machines Lab (revision
ed40e7da625f594fc9c4990f5f66be24873c22d7) - Uncensored edit and AD-Q4_K source: AtomicChat/Inkling-2-Mini-Uncensored-GGUF, AtomicChat (revision
40886ab636cabb4decf4998224f790b462ab048e) - Runtime: llama.cpp, ggml-org and contributors
Citation
@misc{thinkingmachines2026inkling2mini,
author = {{Thinking Machines Lab}},
title = {Inkling 2 Mini},
year = {2026},
url = {https://huggingface.co/thinkingmachines/Inkling-2-Mini},
note = {Announcement: https://thinkingmachines.ai/news/inkling-2-mini/}
}
@misc{atomicchat2026inkling2miniuncensored,
author = {{AtomicChat}},
title = {Inkling-2-Mini-Uncensored-GGUF},
year = {2026},
url = {https://huggingface.co/AtomicChat/Inkling-2-Mini-Uncensored-GGUF}
}
@misc{saben2026looping3090,
author = {Clark Saben},
title = {Looping on a 3090},
year = {2026},
url = {https://github.com/csaben/llama.cpp/tree/inkling-3090},
note = {Write-up link to follow}
}