license: mit
base_model:

• dealignai/DeepSeek-V4.1-Flash-UNCENSORED-FP8
• deepseek-ai/DeepSeek-V4.1-Flash
  base_model_relation: quantized
  tags:
• deepseek
• deepseek-v4.1
• moe
• quantized
• 2-bit
• ternary
• mixed-precision
• gptq
• uncensored
• strata
• xolotl
• consumer-gpu

Update 2026-10-17: the engine is now Xolotl (Strata-DSV4.1 renamed;
old links redirect) - one tree that also serves GLM-5.3-Flash and the Qwen3.8 models. Measured through the server
with thinking on (32K budget): HumanEval 21/21, tool calls 10/10, agentic 3/3, decode 26.8 tok/s. Agent turns got
cheaper: small-turn prefill -12% (bit-identical logits) and a conversation-adaptive expert arena (on by default)
that moves the conversation's NVMe experts into RAM between turns: -21..-30% NVMe reads, a +180-token turn on a 6K
context 8.5 -> 7.6 s. Build with build-xolotl (below).

Update 2026-10-15: the engine's defaults are now the coding-safe route bias ×0.5 (HumanEval 20/21 and tool calls
10/10, vs 16/21 at the old full bias) with DSpark speculative decoding in the server (exact speculative sampling),
a confidence-gated verify (prose 9.5 → 17.3, code 18.8 → 25.5, python 26.6 → 29.1 tok/s), a bit-exact 4-bit KV cache
(--kv-fp4, +1.1 GiB VRAM at 512K) and ~2× faster conversational turns (+1000-token turn 40 → 20 s at 512K).
Details: the repo README "Latest".

Upload complete (2026-10-13): all 16 model parts, V41-MTP-Vision.gguf and the small files are on the Hub.
SHA256SUMS lists every part; reassemble.sh refuses to join until all of them are present and intact.

Experimental research release from a single developer, measured on one PC.
This file is not a llama.cpp / Ollama / LM Studio GGUF. It uses per-expert quant types those runtimes don't load.
Use it with Xolotl (formerly Strata-DSV4.1).

What this is

DeepSeek-V4.1-Flash has 40 layers with 384 routed experts each
(top-6 per token), a shared expert, Hyper-Connections, CSA sparse attention, and an Engram n-gram memory. Its
15,360 routed experts hold 543.6B weights; even at their native FP4 they're far bigger than any consumer PC's RAM.

This repository holds a calibrated mixed-tier quantization of
dealignai's abliterated checkpoint, in which
every expert's gate/up and down projections get their own format:

The dense half (attention, shared expert, output head) is Q8_0. The embedding, compressor and indexer stay F16,
and norms, router and mHC stay F32.

The total is budget-neutral: exactly the bytes of a uniform 2.25-bit (all-Q2_0) quant, but spent where calibration
says they matter. Nearly half the expert weights are ternary, which pays for 3-bit on the experts that need it. On the
calibration proxy, the mix has 17% less output error than uniform Q2_0 at the same size.

"GSQ" names the method family this follows: the format grids and the plan come from
GSQ (Gumbel-Softmax Quantization) and
RCO (budget-exact allocation) by IST Austria's DAS Lab. This release is the
GPTQ-initialized stage of that pipeline with our own "RCO-lite" allocation. GSQ's Gumbel-softmax refinement
hasn't been applied yet; see Not done yet below.

Why

We wanted a frontier-class open model running locally, on hardware people actually own, without slicing it down to a
distilled toy. The design question was where the bits matter. A MoE spends most of its parameters on experts that
each token touches only rarely, so we spend the bit budget where the calibration says it counts, and run the rest
as cheaply as the quality numbers allow.

The engine then treats the experts as a three-tier cache:

How it was made

The whole pipeline is open source in
Xolotl/tools/v41. It runs on a single
RTX 4090.

0. Weight-only metrics can't rank these experts. V4.1's routed experts were trained in FP4, so all 384 experts of a
layer share the same per-32 weight distribution. Every expert in layer 0 has the same round-to-nearest error to three
decimals (ternary 0.443 / Q2 0.378 / Q3 0.183 relative). Which experts deserve more bits can only come from
activations: what the experts actually compute on real text.

1. Calibration corpus (agentic-first). Real coding-agent session transcripts (secrets scrubbed value by value, tool
results clipped) plus antirez/ds4's 4,690 imatrix prompts, all re-rendered with DeepSeek's own V4.1 chat encoder (DSML
tool-call format). That's 128 sequences × 2,048 tokens, with about a quarter of each domain held out: 98 sequences
fit the quantizer, and 30 only ever measure it.

2. A layer-streaming teacher forward. DeepSeek's own reference model.py runs one layer at a time straight
from the HF shards, keeping the hidden state on disk between layers. That makes the teacher the real model, not an
approximation. Each layer's MoE runs expert-major: every FP4 expert is dequantized once and sees all of its routed
tokens.

3. Per-expert GPTQ at three exact grids. For every expert, gate+up are fitted on the Hessian of its input, and down
on the Hessian of its clamped-SwiGLU × routing-weight input. Scales are chosen before GPTQ by a diag(H)-weighted
search (11 clip ratios × both signs) and rounded to what each format stores, so every result is exactly representable
in its GGUF type. GPTQ (damp 1.0) then picks the integers. It beat round-to-nearest in every group (error ratios
0.58 ternary / 0.70 Q2 / 0.74 Q3). Each expert's error is measured on the held-out sequences only, as output error
with the routing weight included.

4. Budget-exact allocation ("RCO-lite"). The cost of a choice is held-out output error divided by the layer's
routed output energy, so every layer counts equally; the first two and last three layers count double. Gate+up and
down are allocated separately. The budget is exactly the all-Q2_0 byte count, solved by Lagrangian relaxation
(bisection on λ) plus a greedy pass to spend the remainder. Guard rails: at most 50% ternary per layer, and experts
with fewer than 8 held-out tokens are floored at Q2_0, so an expert can't drift to ternary without evidence.

5. Sequential quantize pass. A second layer-by-layer chain quantizes every expert at its chosen format. Each layer
is fitted on the outputs of the already-quantized layers below it, so later layers adapt to the errors earlier
ones actually make.

6. Bit-exact export. Block encoders write the GGUF bytes directly and are checked bit for bit against ggml's
dequantizers and the PTQ1_0 reference. Ternary packs 5 trits per byte (28 bytes per 128 weights), and Strata runs it
natively on the CPU and in its grouped-expert CUDA kernels.

Where the bits went

Measured on the held-out calibration tokens; the relative output error is each layer's quantized routed output against
the FP4 original, in the final sequential pass.

Not done yet

• GSQ refinement: Gumbel-softmax optimization of the grid assignments at each expert's chosen format. This is
  the step that closes most of the gap to vector quantization in the GSQ paper. It slots in where GPTQ runs and
  writes the same format.
• A Hadamard rotation for the ternary tier.
• The full RCO manifold search: the allocation above is its Lagrangian "lite" version.

Quality

Perplexity on the 49,146 held-out calibration tokens (never used for fitting): the quantized model vs the original
checkpoint ("teacher"):

Top-1 token agreement: 0.674 (GSQ) vs 0.701 (teacher). In the engine, a 4096-token code prompt gives ppl 2.380 and top-1 0.822.

Prompt processing is exact: every prefill speed-up in the engine keeps the last-token logits byte-identical.
Decode uses two selection-only routing biases by default: a bonus (+1.0) for VRAM-resident experts and a penalty
(−0.6) for NVMe ones. The mixing weights stay exact and every expert stays reachable, but the bias changes which
experts compute each token, and it costs more quality than perplexity suggested (code 2.38 → 2.87). On DwarfStar's
112 official DeepSeek-API fixtures (teacher-forced over the API's own continuations, 6,922 positions; scorer in
tools/v41/quality/):

(102 short prompts; the 10 long prompts route exactly either way: 94.1%.) For reference, DwarfStar's own
IQ2_XXS-Q2_K build scores 89.9% / 95.0% on the same short / long groups. The default bias also causes repetition
loops on some prompts under greedy decoding. Pick the bias for your use: speed-first = default, quality-first = 0.25.
Smarter biases that only bend near-ties were measured and lose to a flat scale.

On real tasks (21 HumanEval problems + 10 tool-call cases, greedy, through the server): exact-ish x0.25 20/21,
x0.5 20/21 (the default since 2026-10-15), x0.75 19/21, full bias 16/21 (the extra failures are logic slips on easy
problems); tool calling 10/10 at every setting.

Against the DeepSeek API (same tasks, paired)

How far is this quant from the real thing? Both sides get the same prompts at temperature 0. "API" is DeepSeek's own
deepseek-flash endpoint (full precision, censored, default reasoning effort); "local" is this quant through the
Xolotl server with the default x0.5 routing bias.

Gauntlet v2 (7 multi-turn agentic coding tasks in a sandbox: the model reads, edits and runs code with tools, then
hidden tests grade every turn; tools/gauntlet/):

The two don't fail the same tasks. The API solved two that the quant got most of the way through (97% and 77% of the
hidden tests passing), and the quant solved one the API didn't (the API reached 91%).

LiveCodeBench v6 (one shot, 32K thinking allowance, the same prompt as the API run). The API scored 90.0% on our
60-problem slice (54/60). Locally we capped the run at one hour, and only 3 problems finished:

3 problems is far too few to put a number on the gap. What it does show:

• The quant still solves hard competitive-programming problems, including one the API got wrong.
• It thinks 3-12x longer to get there. On 3 of the 4 problems it started, the thinking ran into the server's
  27.6K cap and was wrapped up there; it passed both of the finished ones anyway. At ~25 tok/s that is 5-20 minutes
  per problem.
• In agentic work this doesn't show up: total tokens on the Gauntlet are about the same as the API's.

We don't know yet how much of the extra thinking comes from the quantization, the abliteration or the routing bias.

Caveats: small samples; this model is quantized and abliterated; the API's model may not be the exact checkpoint
this was made from.

Speed (RTX 4090 24 GB · Ryzen 7 5700X · 90 GB DDR4 · NVMe)

How each number was reached, step by step:
V41_PROGRESS.md.

Files

Quick start (4 steps)

You need: Linux, an NVIDIA GPU with 24 GB, ~90 GB RAM (the RAM arena is 60 GiB by default; --arena-gib lowers
it, at some speed cost), the CUDA 13 toolkit, and fast SSD space for the model (163 GB) + the Engram tables
(203 GB) = ~366 GB, plus another ~203 GB temporarily while the Engram tables are extracted (or download the
Engram shards to a different drive).

1. Download this repo and join the model (~163 GB):

hf download IndrasMirror/DeepSeek-V4.1-Flash-GSQ-Uncensored --local-dir V41-GSQ
bash V41-GSQ/reassemble.sh V41-GSQ                 # needs ~163 GB free on top of the parts
# or: bash V41-GSQ/reassemble.sh V41-GSQ --delete-parts   (deletes each part once appended: ~163 GB total)

2. Download the two Engram shards - they are NOT in this repo (the …NEED-TO-DOWNLOAD-ENGRAM.txt files mark
them; 101.5 GB each). V4.1 reads its Engram n-gram memory from disk at run time. The tables are unchanged FP8 rows
from the original checkpoint (the abliteration left them byte-identical), so there is no point hosting a third copy:
get them from dealignai/DeepSeek-V4.1-Flash-UNCENSORED-FP8
or deepseek-ai/DeepSeek-V4.1-Flash (same two files).

hf download dealignai/DeepSeek-V4.1-Flash-UNCENSORED-FP8 model-00047-of-00048.safetensors \
   model-00048-of-00048.safetensors model.safetensors.index.json config.json --local-dir FP8-engram

3. Build the engine and extract the Engram tables (one table per fast drive is ideal: decode reads ~48 random
256-byte rows per token):

git clone https://github.com/Indras-Mirror/Xolotl xolotl && cd xolotl
bash tools/xolotl/fetch-llama.sh   # llama.cpp at the pinned commit + the Xolotl ggml patch -> third_party/llama.cpp
# CUDA build, sm_89 (RTX 40xx) shown - see the repo README for other GPUs
cmake -S . -B build-xolotl -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON -DSTRATA_GGML_CUDA=ON \
      -DSTRATA_MMQ_KQUANTS=ON -DCMAKE_CUDA_ARCHITECTURES=89 -DSTRATA_GGML_DIR=$PWD/third_party/llama.cpp -DSTRATA_DSV41_FQ_OP=ON
ninja -C build-xolotl ds41_generate
python3 tools/v41/extract_engram.py ../FP8-engram --to layers.1=/fast/a --to layers.14=/fast/b
# when it has finished (it checks sizes and spot-checks 2000 rows per tensor), ../FP8-engram can be deleted

4. Run (the first load takes ~50 s: it pins the 60 GiB RAM arena):

STRATA_NO_LARGEPAGES=1 build-xolotl/ds41_generate -m ../V41-GSQ/V41-GSQ.gguf --ids-file prompt.i32 -n 128 \
     --backend cuda --experts gpu --engram "/fast/a:/fast/b" --rank ../V41-GSQ/rank.bin

--ids-file takes int32 token ids. For chat and agents, run the OpenAI- and Anthropic-compatible server with tool
calls: python3 tools/v41/serve/serve_ds41.py --engine strata --config tools/v41/serve/strata-ds41.json --port 8150
(see the Xolotl README); it uses
V41-GSQ/tokenizer/.

Credits

• DeepSeek for DeepSeek-V4.1-Flash (MIT).
• dealignai for the abliterated FP8 checkpoint this is quantized from (MIT).
  This release inherits that abliteration: the model has no refusal behaviour, and you are responsible for how you
  use it.
• GSQ (Dadgarnia, Tabesh, Nikdan, Helcig, Kurtić, Kleinegger, Alistarh) and
  RCO (Helcig, Alistarh), IST Austria DAS Lab, for the quantization grids
  and the budget-exact allocation idea this pipeline follows.
• antirez/ds4 for its 4,690 imatrix calibration prompts (re-rendered for V4.1).
• llama-tq for the PTQ1_0 ternary block format (type 143) and its reference kernels (MIT).
• Strata (Niko1221), the tiered MoE engine this work builds on;
  ggml / llama.cpp for the compute backend.
• Quantization pipeline, engine port and measurements: Indra's Mirror (Xolotl, formerly Strata-DSV4.1).

Support

If this has been useful to you and you want to help support what I do, you can shout me a coffee on Ko-fi. Every bit helps keep this going :)