license: mit
base_model:

• deepseek-ai/DeepSeek-V4-Flash-0731
• puwaer/DeepSeek-V4-Flash-0731-reap-150b
• MorinoNushi/DeepSeek-V4-Flash-0731-heretic-abliterated-v2-GGUF-lora
  tags:
• deepseek
• deepseek-v4
• moe
• gguf
• llama.cpp
• turboquant
• tq3_4s
• iq3_xxs
• quantized
• 256k
• agentic
• tool-use
• coding
• reasoning
• uncensored
• abliterated
• heretic
• lora
  language:
• en
• zh
• multilingual
  pipeline_tag: text-generation
  library_name: llama.cpp

DeepSeek-V4-Flash-0731 REAP-150B · TQ3_4S-Container (v3b recipe) · 256K ctx

A custom, selectively-quantized GGUF of the DeepSeek-V4-Flash-0731 architecture, pruned to the
150B "REAP" variant, quantized with a hand-tuned per-tensor recipe and wrapped in the
TurboQuant TQ3_4S container from the turbo-tan/llama.cpp-tq3 fork.

This is not a plain uniform quant. It is the result of a multi-week iterative quantization
campaign on a 3× Turing GPU box, and the recipe reflects a specific, measured trade-off:
maximize context length (256K) while holding wikitext-2 perplexity within noise of the
IQ3_XXS baseline that proved to be the local optimum for this model.

Highlights

What this build buys you over the stock REAP-150B release, in one screen:

• Quality on par with the base REAP model. wikitext-2 perplexity ~13.34, within noise of
  the IQ3_XXS baseline (13.3073 ± 0.23) — the aggressive 3.49 BPW quantization costs no
  measurable quality. (The failed 2-bit IQ2_S experiment measured 14.30, +7.5%, confirming
  this recipe sits on the size/quality optimum, not past the cliff.)
• Uncensored, lossless. A 2 MB rank-1 refusal-ablation LoRA (heretic v2 t256-r2) drops
  refusals from 99.29% → ~7% with low KL drift (0.085), applied via --lora at load. Base
  weights are never modified or requantized — ship it censored or not, same 61 GiB file.
• 2.7× the context. 256K context (vs the 96K ceiling of prior q8_0-KV builds) via the
  TurboQuant q8_0 K + turbo4 V-cache — a 4-bit compressed KV value cache that frees the
  headroom at zero quality cost.
• ngram-mod speculative decoding. ngram-mod (n16/m24) speeds code / structured output to
  ~85 tok/s (100% draft acceptance on repetitive text) over a ~21 tok/s dense baseline.
• Compact VRAM footprint. 61 GiB at 3.49 BPW fits full GPU offload in ~70 GiB — three
  Turing cards (1× 22 GiB + 2× 24 GiB). A uniform Q4_K of the same model is ~84 GiB and
  wouldn't fit; this selective recipe is why it does.
• Ships complete. Everything to serve it end-to-end is in this repo: the model, the uncensor
  adapter, the exact per-tensor recipe, and the importance matrix (see §2a).

1. Summary

The file was quantized from a full-precision Q8_0 re-quantization of the FP8 source
(dsv4-fp8-q8_0.gguf, 79.2 GiB) rather than from the raw FP8 weights, using
--allow-requantize with an importance matrix.

2. Why "TQ3_4S" but IQ3_XXS tensors? (Read this before you assume anything)

The naming is a deliberate container/recipe distinction that trips people up:

• TQ3_4S is the container type. It is the TurboQuant "four-scale" format and the
  general.file_type stamped into the GGUF header. It is what makes stock llama.cpp refuse
  to load this file — you must use the turbo-tan/llama.cpp-tq3 fork (or a build with
  TQ3_4S support) to run it.
• The actual per-tensor types are a recipe, not a uniform TQ3_4S:

The result is that the model is smaller than a uniform Q4_K (which would be ~84 GiB) while
holding quality, because the aggressive IQ3_XXS/IQ3_S quantization is applied only where the
measured error cost is lowest (the deep MoE experts), and the attention path is deliberately
left at Q6_K.

2a. Repository contents (this bundle)

This repository ships everything needed to serve the uncensored model end-to-end, not just
the base quant. Files and their roles:

The base weights and the adapter are separate files for a reason. The uncensoring is a
lossless overlay — the 61 GiB base is never modified or requantized, and the 2 MiB LoRA is
applied in f32/f16 compute regardless of base quant. That means you can ship the base alone
(stock quality, ~99% refusal) or with the adapter (~7% refusal) with no new quantization.
Both files are required only if you want the uncensored behavior; the base is self-sufficient.

Other heret adapter variants (t264/t265/t273) are not bundled here — see §5 for why
t256 is the correct choice for this box and how to fetch the others if you want the
stronger-decensor/slower trade-off.

3. Quantization recipe (exact tensor_types.txt)

token_embd\.weight=q8_0
output\.weight=q8_0
output_hc_fn\.weight=f32
blk\.[0-9]+\.hc_(attn|ffn)_(fn|base|scale)\.weight=f32
blk\.[0-9]+\.ffn_(gate|down|up)_shexp\.weight=q8_0
blk\.[0-9]+\.attn_[a-z_]+\.weight=q6_K
blk\.[0-9]+\.indexer[._][a-z_]\.weight=q6_K
blk\.([0-2]|4[12])\.ffn_(gate|down|up)_exps\.weight=mxfp4
blk\.[0-9]+\.ffn_down_exps\.weight=iq3_s
blk\.[0-9]+\.ffn_(gate|up)_exps\.weight=iq3_xxs

Command used

llama-quantize \
  --allow-requantize \
  --imatrix imatrix-filtered.gguf \
  --tensor-type-file apex-dsv4-v3b.txt \
  --output-tensor-type q6_K \
  --token-embedding-type q6_K \
  dsv4-fp8-q8_0.gguf \
  apex-v4-tq34s-v3b.gguf TQ3_4S 32

• Importance matrix: imatrix-filtered.gguf (222 MiB), 129 entries over 812 calibration
  chunks, generated against the REAP-150B architecture (not the full 284B), so the importance
  data is matched to this pruned variant.
• --allow-requantize is required because the source is already-quantized Q8_0 (itself a
  re-quant of the FP8 release weights).

4. Evaluation

Perplexity (wikitext-2)

The PPL gate for this line of experiments was ≤ 13.44 (the IQ3_XXS baseline 13.3073 ± 0.23
plus a 1% tolerance). This recipe reproduces the v3b recipe exactly, so it lands at the same
~13.34 as the already-validated IQ3_XXS/IQ3_S selective build.

For context on why the experts sit at IQ3_XXS rather than lower, the aborted IQ2_S-expert
candidate (the turbo-tan "champion" recipe) measured 14.30 ± 0.25 — a clear +7.5% regression
that confirms the 2-bit cliff for this pruned model. The IQ3_XXS/IQ3_S combination is the
measured local optimum on the size/quality curve.

⚠️ These are our internal wikitext-2 measurements, not a full Open LLM Leaderboard / MMLU
run. Treat them as relative (this build vs. sibling builds), not absolute capability claims.

Throughput (3× Turing, see §6)

Context length (measured)

The headline win over prior builds is 256K context at the same quality — a 2.7× gain over the
96K ceiling of the previous q8_0-KV build — achieved purely by switching to the compressed
turbo4 V-cache (the TurboQuant fork's 4-bit KV value cache).

5. How to run

Requirements

• The turbo-tan/llama.cpp-tq3 fork (stock llama.cpp will not load TQ3_4S).
• CUDA build: cmake -B build -DGGML_CUDA=ON -DGGML_CUDA_FA=ON -DCMAKE_BUILD_TYPE=Release,
  then cmake --build build -j --target llama-server.
• ~70 GiB combined VRAM for full GPU offload at 256K (see §6 split).

Minimal serving command

CUDA_DEVICE_ORDER=PCI_BUS_ID \
CUDA_VISIBLE_DEVICES="<2080Ti>,<Titan-A>,<Titan-B>" \
llama-server \
  -m apex-v4-tq34s-v3b.gguf \
  -ngl 999 \
  --ctx-size 262144 \
  --cache-type-k q8_0 --cache-type-v turbo4 \
  --flash-attn on \
  -ts 18,22,24 \
  --batch-size 1024 --ubatch-size 512

Recommended full config (agentic / coding)

  --spec-type ngram-mod --spec-ngram-mod-n-max 16 --spec-ngram-mod-n-match 24 \
  --temp 0.65 --top-p 0.90 --top-k 40 --min-p 0.15 \
  --repeat-penalty 1.02 --repeat-last-n 256 --presence-penalty 0.0 \
  --reasoning on --reasoning-format deepseek --reasoning-budget 8192 \
  --lora ds4-flash-heretic-f4-t256-lora-r2.gguf

Chat template / reasoning

The model emits a native reasoning_content field with an inline response marker
(DeepSeek V4 format). The --reasoning-format deepseek flag routes thinking into
reasoning_content and the final answer into content.

Uncensoring (heretic v2 LoRA overlay)

Base deepseek-ai/DeepSeek-V4-Flash-0731 refuses 99.29% of a 140-prompt harmful
eval set. The MorinoNushi/DeepSeek-V4-Flash-0731-heretic-abliterated-v2-GGUF-lora
adapter ds4-flash-heretic-f4-t256-lora-r2.gguf is a rank-1 directional-ablation
LoRA (2 MB) that reduces refusals to ~7% with low KL drift (0.085). It is
lossless: base weights are never modified or requantized, applied in f32/f16
compute regardless of base quant via --lora.

Why t256-r2: it is the only adapter with zero routed-expert tensors
(ffn_down_exps), which sidesteps the known llama.cpp LoRA slowdown (untagged
adapter buffers → host↔device copies + non-quantized mul_mat_id fallback that
disables CUDA graphs). Measured on this box: 21.06 t/s with the LoRA vs 21 t/s
without — no spec-decode penalty. (t264/t265/t273 touch 50-62 expert tensors and
cost a real 10-25% slowdown but reach 3.5-8.6% refusal.)

The adapter is stored alongside the model at
/mnt/data/deepseek-reap-apex/ds4-flash-heretic-f4-t256-lora-r2.gguf and wired
into profiles.json (lora_path), injected by cluster-mode.sh get_profile_args.
Validated 2026-08-28: full compliance on refusal probes, ngram-mod still active.

6. Hardware and the tensor split

Developed and validated on a 3× NVIDIA Turing box:

• Tensor split -ts 18,22,24 (2080Ti=18, Titan=22, Titan=24) was determined by exhaustive
  headroom measurement and is the proven-optimal split for this 61 GiB model.
• CUDA_DEVICE_ORDER=PCI_BUS_ID + explicit CUDA_VISIBLE_DEVICES pinning is mandatory.
  Without pinning, CUDA enumerates the Titans before the 2080 Ti and the split lands on the
  wrong physical cards, causing OOM.
• Host RAM: 60 GiB. Full GPU offload is used (-ngl 999); the model does not spill to CPU.
• 256K context leaves ~231 MiB headroom on the tightest card. 512K degrades (loses pipeline
  parallelism), 1M OOMs.

If your cards are larger (e.g. 3× 48 GiB), 512K becomes clean and 1M is within reach.

7. Pitfalls (learned the hard way)

1. Stock llama.cpp will not load this. general.file_type = 45 (TQ3_4S) is rejected by
   upstream. Use the llama.cpp-tq3 fork.
2. CUDA device enumeration order matters. Without CUDA_DEVICE_ORDER=PCI_BUS_ID pinning, the
   -ts 18,22,24 split maps to the wrong physical GPUs and you OOM even though the total VRAM
   is sufficient.
3. Do not requantize the experts down to 2-bit. IQ2_S experts cost +7.5% PPL on this pruned
   variant (the "champion" recipe does not survive the REAP prune). IQ3_XXS/IQ3_S is the floor.
4. Do not requantize mxfp4 experts up to Q4_K. MXFP4 is denser than Q4_K (4.0 vs 4.58
   bpw); "re-quantizing to Q4" makes the file bigger and lossier. Keep native mxfp4 edges as-is.
5. The DSML tool-call format changed between forks. The old build-deepseek fork emitted
   tool-call text in content; the TQ3 fork emits native OpenAI tool_calls deltas. Any
   middleware written against the old fork must forward native tool_calls deltas or you get
   empty responses + retry loops.
6. Reasoning effort interacts with context. High reasoning effort on trivial prompts causes
   multi-minute think-a-thons. reasoning-budget 8192 + --spec-ngram-mod clamps this.
7. ngram-mod only accelerates structured output. It is ~neutral during the reasoning phase
   (draft acceptance ~18%) and highly effective once the model emits code/structured text
   (up to 100% acceptance).

8. Intended use & limitations

Intended: agentic tool use, long-context coding and analysis, reasoning-heavy workflows,
serving a 150B-class MoE on consumer hardware with a large context window.

Limitations:

• This is a pruned (REAP) variant, not the full 284B model; some capabilities present in the
  full model are reduced.
• Quantization is lossy; for maximum fidelity prefer the FP8 or a BF16 base at the cost of ~2×
  file size and VRAM.
• Requires a non-stock llama.cpp build (the TQ3_4S container).
• Full 1M context is not reachable on 3× Turing (compute-buffer OOM); 256K is the validated
  ceiling there.

9. Reproducibility

1. Obtain the REAP-150B source (Q8_0 re-quant of the FP8 release).
2. Build turbo-tan/llama.cpp-tq3 at 47635d7 with CUDA.
3. Generate or reuse an importance matrix matched to the REAP-150B architecture.
4. Run the quantize command in §3 with the exact tensor_types.txt recipe.
5. Verify: 1328 tensors, 61.01 GiB, 3.49 BPW, general.file_type = 45.

10. Acknowledgements

• deepseek-ai for the DeepSeek-V4-Flash-0731 architecture.
• puwaer for the REAP-150B prune and the FP8/Q8_0 source.
• turbo-tan for the llama.cpp-tq3 fork (TurboQuant TQ3_4S + deepseek4 arch support).
• MorinoNushi for the heretic-gguf directional-ablation tooling and the v2 LoRA adapters
  used for the lossless uncensoring, and p-e-w for the underlying Heretic method.
• The quantization campaign, recipe, uncensoring, and validation were performed by TinkerStack with automated tooling; the build is a community-style release for research and
  self-hosting.