language:
- en
license: apache-2.0
base_model: thinkingmachines/Inkling-2-Mini
base_model_relation: quantized
tags: - gguf
- quantization
- sakura
- imatrix
- uncensored
- moe
- image-text-to-text
Sakura · Inkling 2 Mini Uncensored (GGUF)
Independent community quantization of a modified version of thinkingmachines/Inkling-2-Mini, with reduced refusal behaviour. Not an official release of Thinking Machines.
Needs llama.cpp b12600 or newer (the release these files were made and tested with; it contains the looped Inkling 2 Mini architecture). Older builds, and Ollama or LM Studio versions that bundle an older llama.cpp, will not load these files.
What is this
Inkling 2 Mini is a multimodal looped mixture-of-experts model: 26.4 B parameters, 3.4 B active per token, 128 experts with 8 active, image input, up to 1 M tokens of context. This repository holds smaller versions of a modified variant that refuses far less often than the original. The vision projector (mmproj) is included for image input.
Which file
| File | Quant | Size | Mean KLD vs Q8_0 | Same top token | Refusals (of 40 prompts) |
|---|---|---|---|---|---|
Sakura-Inkling-2-Mini-Uncensored-Q4_K_M.gguf |
Q4_K_M | 17.48 GB | 0.0251 | 92.60 % | 5 / 40 |
Sakura-Inkling-2-Mini-Uncensored-Q5_K_M.gguf |
Q5_K_M | 19.91 GB | 0.0139 | 94.49 % | 3 / 40 |
Sakura-Inkling-2-Mini-Uncensored-Q6_K.gguf |
Q6_K | 23.49 GB | 0.0084 | 94.49 % | 6 / 40 |
Rule of thumb: Q4_K_M if memory is tight, Q5_K_M as the balanced choice, Q6_K for the closest result.
Refusal check: 40 prompts from the public test set mlabonne/harmful_behaviors, greedy decoding, thinking off, 48 tokens per answer; counted are answers that contain a refusal phrase. The unmodified model (ggml-org's IQ4_NL file) refused 16 / 40 of the same prompts. That is a small sample and a phrase count, not a safety evaluation: single answers can still refuse, and a missing refusal says nothing about answer quality.
KLD: each file compared with a Q8_0 made from the same modified weights on a short held-out text (5 chunks of 256 tokens: English, code, German, Chinese). It measures only the quantization step, not the difference to the original model.
Use it
llama.cpp (release b12600 or newer):
llama-server -hf webmp3/Sakura-Inkling-2-Mini-Uncensored-GGUF:Q4_K_M -c 8192 --port 8080
# or with local files, including image input:
llama-server -m Sakura-Inkling-2-Mini-Uncensored-Q4_K_M.gguf --mmproj mmproj-Inkling-2-Mini-BF16.gguf -c 8192 --port 8080
Then open http://127.0.0.1:8080 or use the OpenAI-compatible endpoint http://127.0.0.1:8080/v1. To switch off the reasoning phase, send chat_template_kwargs with reasoning_effort set to none.
Ollama (not tested by the author; needs an Ollama version that includes the inkling2_loop architecture):
ollama run hf.co/webmp3/Sakura-Inkling-2-Mini-Uncensored-GGUF:Q4_K_M
or with a Modelfile next to the downloaded file:
FROM ./Sakura-Inkling-2-Mini-Uncensored-Q4_K_M.gguf
PARAMETER num_ctx 8192
ollama create inkling2-mini-uncensored -f Modelfile && ollama run inkling2-mini-uncensored
LM Studio (not tested by the author; needs a LM Studio version whose llama.cpp runtime includes the inkling2_loop architecture): open the Discover tab, search for webmp3/Sakura-Inkling-2-Mini-Uncensored-GGUF, download the file you want, load it with GPU offload as high as your memory allows and a context length of 8192. For image input, keep the mmproj file in the same folder as the model file.
What was and was not measured
- Measured: refusal phrases on 40 prompts (table above), mean KLD and top-token agreement against the Q8_0 of the same weights, and that every file answers coherently.
- Not measured: benchmarks, answer quality compared with the original, long-context behaviour, image input with these files, speed, GPU backends.
- The 4-bit experts of layers 0 to 19 come from NVFP4 in the original checkpoint, a different 4-bit format than the K-quants here, so the 4-bit files carry a second rounding step.
Responsible use
A model with reduced refusals will answer requests the original would decline. You are responsible for what you do with it. The original's model acceptable use policy still describes the terms of the underlying model.
Sources and license
- Original model: thinkingmachines/Inkling-2-Mini, Apache-2.0.
- The modified weights come from IsValorum/Inkling-2-Mini-Abliterated-BF16-GGUF (BF16 GGUF by IsValorum); I did not create them and cannot vouch for how they were made. The importance matrix and the vision projector files are ggml-org's (ggml-org/Inkling-2-Mini-GGUF), reused unchanged; the matrix was computed on the unmodified model.
Verification details
- Source file revision
c66e5f93eb977ba9065dd905df9f9abfa2196918, SHA-25635344297a6fa6a749dd0165b04fd5bf86b98ee4d116feaada972d8fc4816b6cf; matrix SHA-256553662ef0d5e3d047e5753bf8275c26bbdd4306c60ddf6dc9f46c7f49f19452a; refusal prompts at revision01cead01398926d81f7c52bdb790ee8cf77ebba7. - The matrix has entries for 224 of the 226 16-bit weight tensors of the source file; the other 2 (token embedding and output) were quantized without matrix statistics.
- Made with
llama-quantize --imatrix imatrix.gguffrom the llama.cpp release b12600 (revision29b52d398b294509746ec0a3f28948a3107c0213, CPU build). Every file was checked: its tensor types equal the plan of the same tool's--dry-run, its KLD and refusal numbers were measured with the same build, and each upload was read back from the Hub by SHA-256.
| Projector file (unchanged from ggml-org) | Size | SHA-256 |
|---|---|---|
mmproj-Inkling-2-Mini-BF16.gguf |
642 MB | e914997fe07fb2066bfdce855ec7275c15bc2b624e5423149f1027839593d9cb |
mmproj-Inkling-2-Mini-Q8_0.gguf |
346 MB | 233287272ec5ac39b4168677aba54a4c024f202410f5629fac1830e1059f8abb |
Sakura-Inkling-2-Mini-Uncensored-Q4_K_M.gguf:fdd72a5966b04bc3aa022356df767031f63624ae79bf3af3c718e77e559d00a3Sakura-Inkling-2-Mini-Uncensored-Q5_K_M.gguf:ed3443360c67f00aabed2a96f04def17b85988a0b8abd3e908e1daf60ca64e27Sakura-Inkling-2-Mini-Uncensored-Q6_K.gguf:3882c801a93dc7ec174b5521b878b6ca53cc547433f3c6214687cae41d985ea3
中文说明 · 樱花 Sakura(点击展开)
需要 llama.cpp b12600 或更新版本。 更旧的版本,以及内置旧版 llama.cpp 的 Ollama 或 LM Studio,无法加载这些文件。
- 这是 thinkingmachines/Inkling-2-Mini 的一个修改版本(拒答行为减少)的社区量化。修改后的权重来自 IsValorum/Inkling-2-Mini-Abliterated-BF16-GGUF(IsValorum 提供的 BF16 GGUF),我没有自己制作,也无法担保其制作方式;重要性矩阵与视觉投影文件
mmproj来自 ggml-org,原样复用(矩阵是在未修改的模型上计算的)。 - 文件:Q4_K_M(内存紧张时)、Q5_K_M(均衡)、Q6_K(最接近)。图像输入请把
mmproj文件放在模型文件旁。 - 拒答测试: 来自公开测试集
mlabonne/harmful_behaviors的 40 条提示,贪心解码,关闭思考,每条回答 48 个词元;统计含有拒答用语的回答数。未修改的模型(ggml-org 的 IQ4_NL)拒答 16 / 40,各文件的数字见上表。样本很小,只是用语计数,不是安全评估。 - llama.cpp:
llama-server -hf webmp3/Sakura-Inkling-2-Mini-Uncensored-GGUF:Q4_K_M -c 8192 --port 8080;Ollama:ollama run hf.co/webmp3/Sakura-Inkling-2-Mini-Uncensored-GGUF:Q4_K_M(作者未测试);LM Studio:在 Discover 搜索webmp3/Sakura-Inkling-2-Mini-Uncensored-GGUF(作者未测试),上下文 8192,图像输入时把mmproj文件放在同一文件夹。 - 未测量:基准测试、与原始模型相比的回答质量、长上下文、使用这些文件的图像输入、速度、GPU 后端。
- 使用拒答减少的模型需自行承担责任;原始模型的可接受使用政策仍适用。