base_model:
- xtools-at/DeepSeek-V4-Pro-Qwen3.5-4B-UNCENSORED-heretic
base_model_relation: quantized
language: - en
- zh
- ko
- ja
- es
- ru
library_name: gguf
license: apache-2.0
pipeline_tag: image-text-to-text
tags: - gguf
- llama.cpp
- lm-studio
- qwen3_5
- qwen35
- qwen3.5
- hybrid-attention
- linear-attention
- reasoning
- distillation
- deepseek
- unsloth
- mtp
- heretic
- uncensored
- decensored
- abliterated
- 4-bit
- q4_k_m
- text-generation
- image-text-to-text
DeepSeek-V4-Pro-Qwen3.5-4B-UNCENSORED-heretic — GGUF (Q4_K_M)
GGUF conversion of xtools-at/DeepSeek-V4-Pro-Qwen3.5-4B-UNCENSORED-heretic, quantized to Q4_K_M for llama.cpp / LM Studio / Ollama / llama-cpp-python, with a separate F16 vision projector (mmproj) so the multimodal path keeps working.
This is the 4B sibling of the 9B build in Sakura721/DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-GGUF.
| Base (fp) repo | xtools-at/DeepSeek-V4-Pro-Qwen3.5-4B-UNCENSORED-heretic |
| Upstream chain | Jackrong/DeepSeek-V4-Pro-Qwen3.5-4B (SFT+RL distill of Qwen3.5-4B) → Heretic ARA ablation → this GGUF |
| Architecture | qwen3_5 (qwen35 in GGUF) — hybrid linear-attention + full-attention, 1 MTP head |
| Parameters | 4.326 B (text) · 0.34 B (mmproj) |
| Quantization | Q4_K_M · 5.13 bpw · 222 tensors at Q4_K, 35 at Q6_K, 184 at F32 |
| Context | 262,144 tokens native (rope_theta 10,000,000, interleaved MRoPE [11, 11, 10, 0]) |
| Toolchain | llama.cpp build b12182 (fd27b2034) |
| Modalities | text · image · video (vision needs the mmproj file) |
📦 Files in this repository
| File | Size | Precision | SHA256 |
|---|---|---|---|
DeepSeek-V4-Pro-Qwen3.5-4B-UNCENSORED-heretic-Q4_K_M.gguf |
2.59 GiB / 2,783,449,920 B | Q4_K_M (5.13 bpw) | 729f87101cc689add574497ed43ca0a09d80f542643a3fbcfff5a655dfb5b45a |
mmproj-DeepSeek-V4-Pro-Qwen3.5-4B-UNCENSORED-heretic-f16.gguf |
641.3 MiB / 672,424,864 B | F16 projector | 57d73b2c09f3f9eefc6a125abb0344591f3b21d91d570a6baf95e6aa3172aa92 |
Both files are required for vision, only the first one is required for text-only use.
Tensor type distribution (main model)
| Type | Tensors | Size |
|---|---|---|
Q4_K |
222 | 1,622.1 MiB |
Q6_K |
35 | 1,018.2 MiB |
F32 |
184 | 3.7 MiB |
Q6_K covers the sensitive tensors (attn_v, ffn_down, output-side projections) per the Q4_K_M recipe; all norms, ssm_dt, ssm_conv1d, A_log, dt_bias and recurrent-state tensors stay in F32. Total: 441 tensors, including the 4 MTP tensors of blk.32.
🚀 Quick start
llama.cpp — CLI
# text only
llama-cli -m DeepSeek-V4-Pro-Qwen3.5-4B-UNCENSORED-heretic-Q4_K_M.gguf \
-c 8192 -cnv -t 8
# vision (image / video) — both files required
llama-mtmd-cli -m DeepSeek-V4-Pro-Qwen3.5-4B-UNCENSORED-heretic-Q4_K_M.gguf \
--mmproj mmproj-DeepSeek-V4-Pro-Qwen3.5-4B-UNCENSORED-heretic-f16.gguf \
-c 8192 --image ./photo.jpg
llama.cpp — OpenAI-compatible server
llama-server -m DeepSeek-V4-Pro-Qwen3.5-4B-UNCENSORED-heretic-Q4_K_M.gguf \
--mmproj mmproj-DeepSeek-V4-Pro-Qwen3.5-4B-UNCENSORED-heretic-f16.gguf \
-c 16384 -ngl 99 -t 8 --host 127.0.0.1 --port 8080
curl http://127.0.0.1:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"local","messages":[{"role":"user","content":"Prove that sqrt(2) is irrational."}]}'
The chat template is embedded in the GGUF, so /v1/chat/completions and llama-cli -cnv need no extra flags. Reasoning tokens are returned in reasoning_content; the visible answer arrives after the closing </think> token.
LM Studio
Put both .gguf files in the same folder (a models/<publisher>/<model>/ directory works), then rescan. LM Studio detects the mmproj- prefix automatically and offers the model as a vision model; the Q4_K_M file alone also loads as a text model.
Ollama
# Modelfile (text only — the mmproj projector is not loadable through Ollama today)
FROM ./DeepSeek-V4-Pro-Qwen3.5-4B-UNCENSORED-heretic-Q4_K_M.gguf
PARAMETER temperature 0.7
PARAMETER top_p 0.8
PARAMETER top_k 20
PARAMETER num_ctx 16384
ollama create deepseek-v4-pro-qwen35-4b-heretic -f Modelfile
ollama run deepseek-v4-pro-qwen35-4b-heretic "Solve: integral of x^2 sin(x) dx"
llama-cpp-python
from llama_cpp import Llama
llm = Llama(
model_path="DeepSeek-V4-Pro-Qwen3.5-4B-UNCENSORED-heretic-Q4_K_M.gguf",
n_ctx=16384,
n_threads=8,
verbose=False,
)
print(llm.create_chat_completion(
messages=[{"role": "user", "content": "What is 17 * 23? Show your reasoning."}],
max_tokens=512,
)["choices"][0]["message"]["content"])
🧠 Model architecture
| Field | Value |
|---|---|
| GGUF architecture | qwen35 |
| Hidden size | 2560 |
| Layers | 33 (32 transformer + 1 MTP) |
| Attention layout | 3 × linear-attention + 1 × full-attention per group of 4 (full_attention_interval = 4) |
| Full-attention layers | 8 (indices 3, 7, 11, 15, 19, 23, 27, 31) |
| Q / KV heads | 16 / 4 · head_dim 256 |
| FFN | SwiGLU, intermediate 9,216 |
| Vocab | 248,320 (tokenizer.ggml.pre = qwen35, BPE) |
| Special tokens | BOS: none · EOS: `< |
| RoPE | interleaved MRoPE, sections [11, 11, 10, 0], theta 10,000,000, partial rotary 0.25 |
| SSM (linear attn.) | conv kernel 4 · state size 128 · inner size 2560 · groups 16 · time-step rank 32 |
| Tied embeddings | Yes — no separate lm_head; output.weight aliases token_embd.weight |
Because only 8 of 33 layers hold a growing KV cache, long-context cost is low for its class: roughly 20 KiB per token with an F16 KV cache (≈0.6 GiB at 32 K, ≈5 GiB at 262 K, before the recurrent state).
MTP head
Unlike the 9B build, this checkpoint genuinely ships an MTP head: model-mtp.safetensors provides 15 mtp.* tensors, which the converter maps to blk.32.nextn.{eh_proj,enorm,hnorm,shared_head_norm} (4 tensors). Metadata is therefore self-consistent out of the box:
qwen35.block_count = 33qwen35.nextn_predict_layers = 1qwen35.attention.recurrent_layers= 33 entries, with the last twoFalse
No patching was needed, and the file loads cleanly in llama-server (verified, see below). MTP is not exercised by vanilla decoding but is available to MTP-aware builds for speculative decoding.
🖼 Vision encoder (mmproj)
| Field | Value |
|---|---|
| Projector type | qwen3vl_merger |
| Depth | 24 blocks |
| Hidden size | 1024 |
| Image size / patch | 768 / 16 |
| Spatial merge | 2 |
| Position embeddings | 2304 |
| Tensors | 298 |
The projector is exported at F16 on purpose: quantizing below F16 measurably hurts OCR and grounding accuracy, and it only costs ~0.64 GiB. If you need it smaller, llama-quantize mmproj-...-f16.gguf mmproj-...-q8_0.gguf Q8_0 gives ~360 MiB and is generally safe for captioning, but not for fine-grained text reading.
When loading Qwen-VL-class models,
llama.cppmay warn that grounding tasks want ≥1024 image tokens; add--image-min-tokens 1024tollama-mtmd-cli/llama-serverif you see grounding degradation.
⚙️ How this file was produced
# 1. bf16 safetensors -> F16 GGUF (exact, lossless for these weights)
python convert_hf_to_gguf.py xtools-at/DeepSeek-V4-Pro-Qwen3.5-4B-UNCENSORED-heretic \
--outtype f16 --outfile model-f16.gguf
# 2. F16 -> Q4_K_M
llama-quantize model-f16.gguf DeepSeek-V4-Pro-Qwen3.5-4B-UNCENSORED-heretic-Q4_K_M.gguf Q4_K_M 8
# 3. vision projector, separate file
python convert_hf_to_gguf.py xtools-at/DeepSeek-V4-Pro-Qwen3.5-4B-UNCENSORED-heretic \
--mmproj --outtype f16 \
--outfile mmproj-DeepSeek-V4-Pro-Qwen3.5-4B-UNCENSORED-heretic-f16.gguf
Conversion ran with transformers 5.x, torch 2.15 (CPU) and llama.cpp b12182. The converter performs the V-head regrouping required by in_proj_qkv / in_proj_z / in_proj_a / in_proj_b / conv1d / out_proj (32 value heads over 16 key heads) and always writes qwen35.rope.dimension_sections, so the file is self-consistent without patching. Quantized by Sakura721; the upstream fp weights belong to xtools-at.
✅ Verification performed on this build
| Check | Result |
|---|---|
| Tokenizer integrity | 248,320 tokens, 247,587 merges, chat template 9,059 B — byte-identical across conversion and enrichment |
| Lossless re-packaging | 441/441 model tensors and 298/298 mmproj tensors identical in name/dtype/shape |
| Metadata | block_count 33 · nextn_predict_layers 1 · recurrent_layers 33 · 4 × blk.32.nextn.* present |
| Server load | llama-server + --mmproj → loaded multimodal model, GET /health → {"status":"ok"} |
| Text inference | 7*8=? → 56, finish_reason: stop, reasoning trace well-formed |
| Vision inference | 384×384 test image (red frame, white field, navy disc) described correctly over /v1/chat/completions |
Reference speed on the machine used for conversion — i5-9300H (4C/8T), 16 GB RAM, CPU only — with llama-cli:
| Phase | t/s |
|---|---|
| prompt processing | ~22 |
| generation | ~7 |
CPU-only numbers; expect materially higher throughput once the file is offloaded to a GPU with -ngl 99.
🌡 Sampling defaults
A reasonable starting point for this family (adjust to taste):
| Parameter | Value |
|---|---|
temperature |
0.7 (0.6 for math/STEM, 0.9 for creative writing) |
top_p |
0.8 |
top_k |
20 |
min_p |
0.0 |
repeat_penalty |
1.0–1.05 |
The model is a reasoning model: it emits <think> … </think> before the answer. Keep the full reasoning trace when solving math/STEM problems — truncating it visibly degrades accuracy.
📊 Inherited benchmark numbers
These results belong to the unquantized base checkpoint and are reproduced here for context only. They were not re-measured on this Q4_K_M file. Quantization error at Q4_K_M on a 4B model is normally small but larger than at 9B, so treat the numbers as upper bounds.
| Benchmark | 4B base | 9B sibling | Notes |
|---|---|---|---|
| GSM8K | 91.77 % | 94.50 % | 4B: two full runs over all 1,319 test questions (2,638 answers, 2,421 correct), MTP-Q8 + llama.cpp |
| MMLU-Pro Math | 80.40 % | 92.40 % | 500 sampled questions |
| MMLU-Pro Physics | 74.20 % | 89.40 % | 500 sampled questions |
| MMLU-Pro Chemistry | 74.80 % | 89.80 % | 500 sampled questions |
| MMLU-Pro Average | 76.47 % | 90.53 % | 1,500 questions total |
Abliteration quality, from the base repo: 5/100 refusals vs. 99/100 for the unmodified model, KL divergence 0.0378. Heretic Arbitrary-Rank Ablation was applied to the 4B checkpoint before this conversion.
⚠️ Limitations
- Experimental community model. Hallucinations, arithmetic slips and unstable behaviour on underspecified tasks remain possible; verify high-stakes math, science and factual output independently.
- Quantization is not free. 5.13 bpw on a 4B model introduces a real, measurable drop versus bf16 — most visible in long derivations, rare-script OCR and code. Use
Q5_K_M/Q6_K/Q8_0if you need more headroom. - Vision is a bolt-on. The projector is F16 but the language tower is Q4_K_M, so image-conditioned reasoning inherits the same quantization error; small text in images is the first thing to suffer.
- Reasoning traces can be long.
-cmust leave room for<think>; running at 2 K context will truncate reasoning mid-flight. - 4B is smaller than 9B. MMLU-Pro gaps of 8–15 points versus the 9B sibling are real; the 4B wins only on footprint and speed.
- Decensored weights. This model has had its refusal behaviour ablated. It will produce content that aligned models filter, including content that is wrong, offensive or unsafe. You are responsible for how you deploy it and for complying with the licences and laws that apply to you.
- The architecture (
qwen3_5) is recent: use allama.cppbuild from b12182 or newer, and avoid forks that predateQWEN35support — they will refuse to load the file.
📚 Citation
@misc{sakura721_deepseek_v4_pro_qwen35_4b_heretic_gguf,
title = {DeepSeek-V4-Pro-Qwen3.5-4B-UNCENSORED-heretic GGUF (Q4\_K\_M)},
author = {Sakura721},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/Sakura721/DeepSeek-V4-Pro-Qwen3.5-4B-UNCENSORED-heretic-GGUF}},
note = {GGUF quantization of xtools-at/DeepSeek-V4-Pro-Qwen3.5-4B-UNCENSORED-heretic}
}
@misc{jackrong_deepseek_v4_pro_qwen35_4b,
title = {DeepSeek-V4-Pro-Qwen3.5-4B},
author = {Jackrong},
year = {2026},
publisher = {Hugging Face}
}
🙏 Acknowledgements
- Jackrong — the
DeepSeek-V4-Pro-Qwen3.5-4Bdistillation this line descends from. - Heretic — arbitrary-rank ablation tooling used to decensor the base model.
- Qwen — the Qwen3.5-4B architecture and tokenizer.
- Unsloth — fine-tuning tooling used upstream.
- ggml-org / llama.cpp — the converter, quantizer and runtime that made this file possible.
Quantized with llama.cpp b12182 on Windows (CPU-only), from the bf16 safetensors of the base repository.