library_name: vllm
license: mit
base_model:

• deepseek-ai/DeepSeek-V4.1-Flash-1017
• LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-3.5bpw
• pqhaz/DeepSeek-V4.1-Flash-1017-heretic
  tags:
• deepseek
• exl3
• text-generation
• tool-use
  pipeline_tag: text-generation

Visual comparisons: EXL3 3.5-bit experts

Native mixed MXFP4 · NVFP4 target · EXL3 2.875-bit experts · EXL3 3.5-bit experts

Four formats, one checkpoint family. Dark charts show measured behavior, coding, speed, latency and context comparisons; full numeric tables and protocols follow the figures.

Did the edit change harmful-response behavior?

300 HarmBench prompts. Higher green share means more harmful compliance, not a generic refusal rate.

JailbreakBench prompts: chat-mode comparison

100 JBB prompts; auxiliary HarmBench grading, not official JBB scoring. Higher green share means more harmful compliance, not a generic refusal rate.

JailbreakBench prompts: reasoning-mode comparison

100 JBB prompts; auxiliary HarmBench grading, not official JBB scoring. Higher green share means more harmful compliance, not a generic refusal rate.

Capability retention

HumanEval: network-disabled execution, 164 tasks. Strict MMLU: eight-token cap, format-sensitive. Small differences are not proof of superiority.

Coding and harness checks

HumanEval: one greedy completion, no repair, 3 s CPU/wall limit per case, 512 MiB address-space limit. Editing checks are bounded fixtures, not SWE-bench.

Chat: throughput and interactivity

Three measured legs per point; bars show observed min–max, not confidence intervals. Native repeat has measured endpoints only.

Warm agent: throughput and interactivity

Three measured legs per point; bars show observed min–max, not confidence intervals. Native repeat has measured endpoints only.

Reasoning agent: throughput and response latency

Three measured legs per point; bars show observed min–max, not confidence intervals. Native repeat has measured endpoints only.

Cold agent context: concurrency costs

Two measured legs per point; bars show observed min–max, not confidence intervals.

Draft acceptance by target format

Three measured legs. Acceptance is a server-counter metric, not an end-to-end speedup guarantee.

Drafter comparison on the native target

Three measured legs. Drafting-off reasoning was not measured. Same three-stage MTP weights; NVFP4 is a representation/calibration experiment.

Long-input prefill

Three measurements per length. Effective rate is input tokens divided by first-event latency, not an isolated kernel rate. OS file cache was not standardized.

Storage and context

300K total context includes input + output. Thirty-two scheduler slots do not prove 32 simultaneous 300K sessions. Harness fixtures are bounded checks.

Inference settings

Exact flags, package versions, image IDs and cache metadata are in comparison-data.json. Main target quantization differs from the optional NVFP4 drafter.

How these comparisons were measured

These charts compare the same DeepSeek V4.1 Flash 1017 checkpoint family on four RTX PRO 6000 Blackwell 96GB GPUs. They include the original base and alternative edits where measured. Main expert quantization and serving implementations differ; this is a comparison of complete serving configurations, not an isolated quantization experiment.

Reading the behavior charts: harmful-response success means the pinned classifier judged that the final answer complied with the harmful benchmark request. It is not a generic refusal rate. HarmBench uses 300 direct prompts; JailbreakBench rows use 100 prompts with an auxiliary HarmBench rubric, not official JBB grading. The classifier passed 95/100 balanced validation controls. Unknown and empty cases remain visible; output-capped counts are disclosed.

Reading the performance charts: points are means of per-leg metrics; error bars show the observed minimum and maximum, not confidence intervals. Latency panels average each leg's median (or explicitly labeled percentile). Three legs were measured except cold-context waves, which used two. Natural output lengths and unrelated shared-GPU workloads vary. The white native-repeat markers are measured C1/C32 endpoints only.

All four released formats use the native DSpark multi-token prediction (MTP) drafter with five speculative positions. The NVFP4 drafter experiment changes drafting on the native target; it is distinct from the NVFP4 target checkpoint. Its gains are not assumed to transfer to EXL3. Both EXL3 implementations retain native MTP weights.

The coding evidence is executed HumanEval plus two real editing fixtures in each of Claude Code, OpenCode and OMP. HumanEval ran in a network-disabled, read-only container with isolated per-case CPU, memory and time limits. These are function-level coding and bounded integration checks. Repository-level SWE-bench has not been measured.

Native/MXFP4 is one mixed-format baseline. EXL3 bit rates refer to the main routed experts, not the whole model. Runtime-reported KV capacity and 32 scheduler slots do not establish 32 simultaneous full 300K sessions. Source measurements are from October 17–18, 2026.

Numeric data and inference settings

The text tables below contain the values plotted above. Download complete source measurements and exact runtime settings, including all individual legs, image IDs, package versions, server flags, cache metadata, classifier counts and source hashes. Plain-text tables are also available.

The four-format timing sweeps use a 300,000-token total context window and a maximum of 32 active sequences. Chat/warm/cold sweeps use temperature 0, top-p 1 and a 768-token output cap. Reasoning sweeps use temperature 1, top-p 1 and a 16,384-token cap at concurrency 1, 8 and 32. Prefill uses unique public-code prefixes and a short READY reply. Quality and behavior protocols have separate budgets; their exact settings are retained in the detailed text and JSON.

Recreate the figures after downloading the JSON and plotting script:

uv run --no-project --with matplotlib --with numpy python evaluation/plot-comparisons.py --bundle evaluation/visual-comparison.json --out comparison-figures

DeepSeek V4.1 Flash 1017: EXL3 at 3.5 bits per main expert weight

Capability, harness, timing-integrity and exact portable-package checks are complete. Independent behavior results were reviewed and are reported below. These are bounded local tests, not a universal best-method or zero-refusal guarantee.

The main routed experts average 3.5 nominal bits per weight. Dense, attention, shared-expert, MTP and vision weights remain native, with FP8 Engram tables. This is not a whole-model bits-per-weight claim or a copy of the publisher’s complete quantization.

The package contains 188,245 indexed tensors in 88 weight shards, totaling 460,028,516,632 shard-file bytes. File size is not GPU allocation. Vision weights are retained; the qualification below covers text and tool use, not a vision benchmark.

Provenance and numerical path

Base: DeepSeek-V4.1-Flash-1017. The attention edit is pqhaz’s published 38-tensor patch, SHA256 16e63cd3e79c3590855aad340b5b9dc37f27562bf2d0a40f85268915da221ce2. This experiment applied that patch; it did not originate the edit or run the upstream Heretic optimizer.

Calibrated routed experts come from LibertAIDAI’s 3.5-bit release. The hybrid assembly did not recalibrate those codebooks. The runtime uses TP4 plus expert parallelism, native-dense metadata adapters, corrected expert-local dimensions and packed-cache page mappings. SM120 uses 64-row sliding-window pages; calls with 128-row secondary pages are bounded to 64 queries where the larger FlashInfer specialization is unavailable. Runtime source pins and licenses are under runtime/. Rebuilding is not claimed to reproduce the exact Docker image ID. Full capability, behavior and timing measurements belong to image sha256:6094b86f17a7523bdd9c95c291926c68f5071c643bf0b29b543c4fcf60b64804. The recipe rebuilt image sha256:fa130f4e3d8b1dee2e55f098fe466bac3712dab560b11d0679b5e5f95e789b3b passed separate GPU primitives, functional checks and paired decoding acceptance on the 2.875-bit expert pack, starting with an empty mounted compilation cache; the OS file cache was not cleared. All 112 embedded CUDA binaries match byte-for-byte after excluding only debug-line bytes. All 500 installed system-package versions and the checked runtime-library hashes matched in this rebuild. Apt package versions are not pinned, so future rebuilds still need validation. The Docker images are not byte-identical. See evaluation/rebuilt-image-acceptance.json; this acceptance is not a new timing or full-capability run.

Capability retention

The unchanged engineering gates passed. Generation used temperature 0, thinking off and matched prompts/budgets. HumanEval was executed in a network-disabled sandbox. Strict MMLU has an eight-token answer cap and is format-sensitive; it is not an official leaderboard protocol. Error, truncation and paired-discordance details are in evaluation/full-comparison.json. Paired intervals do not include run-to-run numerical variation.

Independent behavior evaluation

The pinned official HarmBench classifier passed 95/100 fixed balanced human-majority controls (45/50 negative and 50/50 positive). Chat HarmBench uses 200 standard and 100 contextual direct prompts. JBB rows are auxiliary use of that classifier on JailbreakBench prompts, not official JailbreakBench scoring. Only final content is evaluated. Empty finals are unsuccessful and remain in the denominator; their automatic handling is separate from model classification. Output-budget exhaustion is not the same as a refusal. These are behavior-success counts, not a generic refusal rate or attack-search result. Coverage and the reviewed decision are in evaluation/behavior.json and evaluation/behavior-review.json. Raw requests and generations are excluded.

Real harness and context checks

Two editing tasks passed in each of Claude Code, OpenCode and OMP. The exact packaged wrappers also passed tool-driven memory fixtures at these observed input lengths:

The configured window is 300,000 total tokens. Input and output share that budget. These fixtures verify retained early/final facts, routing and tool operations; they do not establish general coding or long-context reasoning reliability. evaluation/runtime-qualification.json binds the tested runtime/configuration files and fixture receipts.

Performance and limits

All 69 timing cells completed the fixed integrity checks: 0 API errors, 0 empty answers and 0 output-limit hits across the recorded requests. Per-cell limits and counts are in evaluation/performance.json. Chat, warm-agent and reasoning use three measured legs after priming/warmup; prefill uses three repetitions and cold-context uses two. Outputs stop naturally. Concurrency covers 1, 2, 4, 8, 16 and 32, with a separate thinking sweep at 1, 8 and 32. Cold-context waves and prefill at 8K, 32K, 128K and 285K are separate protocols.

The measurements used four RTX PRO 6000 Blackwell Workstation 96GB GPUs with unrelated training and embedding workloads active. They are shared-hardware results. Different formats use different runtime paths, so these are serving-configuration comparisons, not isolated quantization effects. OS/Engram cache history was not standardized. Thirty-two scheduler slots do not prove 32 simultaneous full-window sessions. Comparisons, plots and protocol code are in the benchmark dataset.

Run

Use Linux with Docker, NVIDIA Container Toolkit, hf, uv and sufficient storage. The EXL3 build additionally uses CUDA 13.3. Run one four-GPU model server at a time. The launch recipe does not stop or replace another container. Pin a repository revision when reproducing measurements.

DS_MODEL_DIR=/data/deepseek-v41-model
DS_CACHE_DIR=/data/deepseek-v41-cache
hf download Infatoshi/DeepSeek-V4.1-Flash-1017-EXL3-3.5bpw-uncensored --local-dir "$DS_MODEL_DIR"
bash "$DS_MODEL_DIR/runtime/build-exl3.sh" "$DS_CACHE_DIR/build" /usr/local/cuda-13.3
bash "$DS_MODEL_DIR/runtime/serve-exl3.sh" "$DS_MODEL_DIR" "$DS_CACHE_DIR" /usr/local/cuda-13.3

The server listens on loopback port 8317 in vLLM development mode, with model ID deepseek-v41-local, a 300K window, native DSpark drafting and disk-backed Engram. The tested image ID is sha256:6094b86f17a7523bdd9c95c291926c68f5071c643bf0b29b543c4fcf60b64804. Rebuild the pinned EXL3 recipe; the recorded local image ID is not a public registry reference.

Install the corresponding client CLIs first. For first-time OMP setup, register the local provider with uv run --no-project python "$DS_MODEL_DIR/runtime/harnesses/configure-omp.py". The helper preserves other providers and refuses to overwrite an existing local-provider entry.

With the server reachable at http://127.0.0.1:8317, use the packaged clients:

bash "$DS_MODEL_DIR/runtime/harnesses/claude-ds.sh"
bash "$DS_MODEL_DIR/runtime/harnesses/opencode-ds.sh"
bash "$DS_MODEL_DIR/runtime/harnesses/omp-ds.sh"

From another machine, forward port 8317 to the GPU host first. The Claude wrapper keeps the normal profile and enables bypassPermissions. Client routing is session-local; global credentials are not copied. The OpenCode wrapper passes the model explicitly to override agent defaults. Large initial text fixtures use stdin rather than assuming file attachments retain every byte.

The OMP wrapper loads omp-context.yml for that invocation. Automatic compaction and mid-turn checks stay enabled, with 24,576 tokens reserved for the 16,384-token output budget plus 8,192 tokens of headroom. At the configured 300,000-token window, the reserve-based maintenance trigger is 275,424 tokens. Global settings and other providers are unchanged.

Weights retain the upstream MIT license. Runtime components retain their included licenses, including the EXL3 disk runtime’s AGPL-3.0 material. release-files.json records file hashes; publication is verified against the immutable Hugging Face commit.