pretty_name: DeepSeek V4.1 Flash 1017 local evaluation
license: mit
task_categories:

• text-generation
  tags:
• benchmark
• vllm
• deepseek

Visual comparisons: All four serving configurations

Native mixed MXFP4 · NVFP4 target · EXL3 2.875-bit experts · EXL3 3.5-bit experts

Four formats, one checkpoint family. Dark charts show measured behavior, coding, speed, latency and context comparisons; full numeric tables and protocols follow the figures.

Did the edit change harmful-response behavior?

300 HarmBench prompts. Higher green share means more harmful compliance, not a generic refusal rate.

JailbreakBench prompts: chat-mode comparison

100 JBB prompts; auxiliary HarmBench grading, not official JBB scoring. Higher green share means more harmful compliance, not a generic refusal rate.

JailbreakBench prompts: reasoning-mode comparison

100 JBB prompts; auxiliary HarmBench grading, not official JBB scoring. Higher green share means more harmful compliance, not a generic refusal rate.

Capability retention

HumanEval: network-disabled execution, 164 tasks. Strict MMLU: eight-token cap, format-sensitive. Small differences are not proof of superiority.

Coding and harness checks

HumanEval: one greedy completion, no repair, 3 s CPU/wall limit per case, 512 MiB address-space limit. Editing checks are bounded fixtures, not SWE-bench.

Chat: throughput and interactivity

Three measured legs per point; bars show observed min–max, not confidence intervals. Native repeat has measured endpoints only.

Warm agent: throughput and interactivity

Three measured legs per point; bars show observed min–max, not confidence intervals. Native repeat has measured endpoints only.

Reasoning agent: throughput and response latency

Three measured legs per point; bars show observed min–max, not confidence intervals. Native repeat has measured endpoints only.

Cold agent context: concurrency costs

Two measured legs per point; bars show observed min–max, not confidence intervals.

Draft acceptance by target format

Three measured legs. Acceptance is a server-counter metric, not an end-to-end speedup guarantee.

Drafter comparison on the native target

Three measured legs. Drafting-off reasoning was not measured. Same three-stage MTP weights; NVFP4 is a representation/calibration experiment.

Long-input prefill

Three measurements per length. Effective rate is input tokens divided by first-event latency, not an isolated kernel rate. OS file cache was not standardized.

Storage and context

300K total context includes input + output. Thirty-two scheduler slots do not prove 32 simultaneous 300K sessions. Harness fixtures are bounded checks.

Inference settings

Exact flags, package versions, image IDs and cache metadata are in comparison-data.json. Main target quantization differs from the optional NVFP4 drafter.

How these comparisons were measured

These charts compare the same DeepSeek V4.1 Flash 1017 checkpoint family on four RTX PRO 6000 Blackwell 96GB GPUs. They include the original base and alternative edits where measured. Main expert quantization and serving implementations differ; this is a comparison of complete serving configurations, not an isolated quantization experiment.

Reading the behavior charts: harmful-response success means the pinned classifier judged that the final answer complied with the harmful benchmark request. It is not a generic refusal rate. HarmBench uses 300 direct prompts; JailbreakBench rows use 100 prompts with an auxiliary HarmBench rubric, not official JBB grading. The classifier passed 95/100 balanced validation controls. Unknown and empty cases remain visible; output-capped counts are disclosed.

Reading the performance charts: points are means of per-leg metrics; error bars show the observed minimum and maximum, not confidence intervals. Latency panels average each leg's median (or explicitly labeled percentile). Three legs were measured except cold-context waves, which used two. Natural output lengths and unrelated shared-GPU workloads vary. The white native-repeat markers are measured C1/C32 endpoints only.

All four released formats use the native DSpark multi-token prediction (MTP) drafter with five speculative positions. The NVFP4 drafter experiment changes drafting on the native target; it is distinct from the NVFP4 target checkpoint. Its gains are not assumed to transfer to EXL3. Both EXL3 implementations retain native MTP weights.

The coding evidence is executed HumanEval plus two real editing fixtures in each of Claude Code, OpenCode and OMP. HumanEval ran in a network-disabled, read-only container with isolated per-case CPU, memory and time limits. These are function-level coding and bounded integration checks. Repository-level SWE-bench has not been measured.

Native/MXFP4 is one mixed-format baseline. EXL3 bit rates refer to the main routed experts, not the whole model. Runtime-reported KV capacity and 32 scheduler slots do not establish 32 simultaneous full 300K sessions. Source measurements are from October 17–18, 2026.

Numeric data and inference settings

The text tables below contain the values plotted above. Download complete source measurements and exact runtime settings, including all individual legs, image IDs, package versions, server flags, cache metadata, classifier counts and source hashes. Plain-text tables are also available.

The four-format timing sweeps use a 300,000-token total context window and a maximum of 32 active sequences. Chat/warm/cold sweeps use temperature 0, top-p 1 and a 768-token output cap. Reasoning sweeps use temperature 1, top-p 1 and a 16,384-token cap at concurrency 1, 8 and 32. Prefill uses unique public-code prefixes and a short READY reply. Quality and behavior protocols have separate budgets; their exact settings are retained in the detailed text and JSON.

Recreate the figures after downloading the JSON and plotting script:

uv run --no-project --with matplotlib --with numpy python evaluation/plot-comparisons.py --bundle evaluation/visual-comparison.json --out comparison-figures

DeepSeek V4.1 Flash 1017: local measurements

Completed qualification and releases. Both native attention edits have completed capability, harness and performance collection. The attention-only edit remains the working reference; the NVFP4 target has completed capability, independent behavior grading, timing and exact-package qualification and is published. EXL3 full capability, harness and all timing collection have passed; independent behavior grading, exact package qualification and separate clean-build GPU acceptance passed. All four serving configurations are published; immutable revisions are in released-models.json. No checkpoint weights or raw generated answers are included here.

Updated October 18, 2026, 2:24 pm Mountain Time. These are local measurements on four RTX PRO 6000 Blackwell Workstation Edition GPUs with unrelated training and embedding workloads active. They are not isolated-hardware results or official DeepSeek leaderboard scores.

Completed native reasoning performance

Warm 40K public-code prefix plus approximately 1K new observations per request. The task asks for a small cache bug fix and two tests. DSpark proposes up to five tokens with adaptive verification. Temperature 1, top_p 1, seed 731, natural stopping, 16,384-token output cap. Three measured legs per cell, after priming and warmup. All nine legs had zero API errors, empty final answers and output-limit hits; request counters matched.

Output tokens include reasoning. Aggregate throughput divided by concurrency is not the same metric as an individual request's decode rate. Latency columns average each leg's median. Bars show observed min–max, not confidence intervals.

results.json also contains completed native chat/warm sweeps at concurrency 1, 2, 4, 8, 16 and 32; independent cold-context waves; and cold-prefix prefill at 8K, 32K, 128K and 285K. Cold means KV-prefix cold. Engram and OS caches were not reset. The runtime reports 4,531,484 KV-cache tokens, approximately 15 full 300K contexts before workload overhead; that is a capacity estimate, not a measured 15-user result.

Completed method comparison

Rank-2 reasoning throughput at concurrency 1, 8 and 32 averaged 124.43, 366.53 and 836.92 output tokens/s; mean leg-median visible-answer latency was 7.20, 19.29 and 39.52 seconds. It did not consistently outperform the attention-only reference across throughput and latency. All reasoning cells had zero errors, empty answers and output caps. Generated answer lengths differ; these are shared-hardware observations.

figures/comparison/ also contains the completed chat and warm-agent concurrency sweeps, cold-context waves and prefill plots. Full checkpoint hashing preceded rank-2's cold-context sweep. OS file-cache state was not standardized, so cold/prefill differences cannot be attributed solely to the attention edit.

Published checkpoints

• Native mixed-MXFP4 checkpoint: 86 files verified, including the qualified per-run OMP context profile. Model/server weights are unchanged from the measured native configuration.
• NVFP4 checkpoint: 92 files verified. Exact packaged Claude/OpenCode/OMP fixtures reached 258,996/281,560/266,240 input tokens with no compaction; all six editing tasks passed.

The OMP profile keeps automatic compaction enabled and reserves 24,576 tokens for 16,384 output tokens plus 8,192 headroom. It is used by the production wrapper, with a 275,424-token reserve-based trigger within the 300K server window. The first NVFP4 attempt compacted under OMP’s default 15% reserve and was retained as a failure; a fresh full qualification passed after this client-only correction. No model weights, benchmark fixtures or qualification gates changed. released-models.json lists immutable release revisions. EXL3 release status is recorded in that same revision index.

Native versus NVFP4 drafting

All these trials use the same attention-edited native target. The optional NVFP4 drafter re-expresses its existing three-stage MTP expert weights and uses four-bit draft activations; it is not a separately trained draft model. Its 36 chat/warm, nine reasoning, 12 cold-context and 12 prefill measurements completed with integrity checks passing. Independent behavior results are reported below. This optional drafter remains an evaluated experiment, not a separately published checkpoint or the selected default.

Reasoning throughput at concurrency 1, 8 and 32 averaged 122.21, 445.09 and 818.89 tokens/s, versus native 131.55, 425.19 and 830.06. Mean leg-median visible-answer latency was 9.10, 16.43 and 33.72 seconds, versus native 9.65, 16.76 and 30.14. There is no consistent speed advantage; native drafting remains the default. Natural answer lengths vary, and unrelated workloads remained active.

Disabling drafting completed all 36 chat/warm cells. Native drafting delivered 1.47–2.08 times their aggregate throughput across the 12 matched workload/concurrency means. No-drafting reasoning, cold-context and prefill sweeps were explicitly deferred to prioritize target-format qualification; no partial measurements are included here. Reported KV capacity increased from 4.53 million tokens with native drafting to 7.30 million without drafting. The NVFP4 drafter reported 4.41 million. These are capacity metadata, not measured full-window concurrency.

The optional drafter passed its 1889-case capability screen, six real harness editing tasks and the long Claude fixture. Its repeated-output diagnostic matched 105/128 outputs, versus a native-repeat floor of 106/128; this is not bitwise parity. Calibration observed 280 of 384 expert inputs. The remaining 104 input scales use normalization bounds, and all intermediate scales use the activation clamp bound. The source and fixed scales are under code/nvfp4-draft/; their overlay hashes match the measured runtime. They retain the included Apache-2.0 license.

After preparing the native edited model as below, the optional launch recipe is:

bash code/serve-nvfp4-draft.sh "$DS_BENCH_WORK/edited" "$DS_BENCH_WORK/cache"

Use the same benchmark commands with a fresh output directory and explicit arm label. The measured private launcher completed qualification collection; this portable wrapper changes mount locations and has passed syntax/source review but has not yet had a standalone GPU boot. That packaging limitation is separate from the completed measurements. Neither launcher stops another server. MODE=off bash code/serve-native.sh ... selects the no-drafting control.

Completed NVFP4 target measurements

The main routed experts use NVFP4 weights and four-bit activations with layer-wide input-scale maxima. The three-stage MTP drafter stays native. All 320 prepared-scale checks passed; FORCE_A16 remained 0. This differs from the optional NVFP4 drafter above.

All 69 timing cells passed: 36 chat/warm, nine reasoning, 12 cold-context and 12 prefill. Chat throughput at concurrency 1 and 32 averaged 101.60 and 796.13 tokens/s, versus native 146.84 and 831.66. NVFP4 trailed native at every chat concurrency. Warm-agent throughput was 1.2–4.2% higher at concurrency 4–32, but lower at 1 and 2. Reasoning throughput at concurrency 1/8/32 was 86.58/363.05/780.69 tokens/s, versus 131.55/425.19/830.06. Native remains the default; this is no global fastest-format claim.

Cold first-event latency at concurrency 1/8/32 was 5.329/27.961/115.868 seconds, versus native 5.182/24.709/104.618. Prefill at 8K/32K/128K/285K was 0.978/3.842/15.518/37.183 seconds, versus 0.976/3.925/15.606/37.875. Means and observed leg ranges are in the figures and JSON. Unrelated GPU workloads remained active. Hashing overlapped behavior generation and upload overlapped thinking-quality generation; both ended before all timing. Harness and long-context checks intervened. OS/Engram cache history was not standardized.

Full capability scores were 1,277/1,319 GSM8K, 155/164 HumanEval and 11,796/14,042 strict MMLU, versus original 1,279/157/11,920 correct respectively. The MMLU change was −0.883 percentage points, within the unchanged one-point gate. Six editing tasks passed, and Claude reached 259,295 input tokens with no compaction. Independent behavior results are reported below; capability scores alone do not certify refusal behavior.

Reported KV capacity was 1,571,012 tokens, approximately 5.24 full 300K contexts before overhead, versus native 4,531,484 and 15.10. These are startup estimates, not measured full-window user counts. Runtime weights plus other non-PyTorch allocations were about 80.8 GiB/GPU versus 76.2 native; this is not an independent memory profile or proof of a solely datatype-caused difference.

The measured image digest, overlay digest, runtime settings and aggregate checks are in results.json. code/serve-nvfp4-layermax.sh and code/nvfp4-target/ preserve the required layer-max setup. The exact packaged runtime and final NVFP4 checkpoint passed qualification and are published; use the verified release recipe linked in released-models.json. Source weights are not included in this benchmark dataset. Reproduction requires the matching serialized NVFP4 target, not arbitrary NVFP4 weights. The native checkpoint is published at Infatoshi/DeepSeek-V4.1-Flash-1017-MXFP4-uncensored, verified revision aab12f7906f4b7d335fb0b6e72d7fd9d788cd95a.

EXL3 main-expert candidate: completed capability and main timing

The calibrated main routed experts average 2.875 nominal bits per weight. Dense, attention, shared-expert, MTP and vision weights remain native, with FP8 Engram tables. This is not a whole-model bits-per-weight claim or a copy of the publisher's complete quantization. Immutable source revisions are in provenance.json.

Full capability gates passed: 1,279/1,319 GSM8K, 158/164 HumanEval and 11,859/14,042 strict MMLU, versus original 1,279, 157 and 11,920. The MMLU change is −0.434 percentage points. Its constrained-choice diagnostic matched native at 1,230/1,425. Six real editing tasks passed across Claude Code, OpenCode and OMP. A real Claude session reached 259,155 input tokens, reported a 300,000-token window, retained the checked early/final facts and had no compaction. These are bounded fixtures; aggregate evidence is in exl3-harness.json.

All 36 chat/warm timing cells completed with zero errors, empty answers, output caps and request-counter mismatches. The table gives mean aggregate output tokens/s across three legs. Shared hardware and natural answer lengths limit attribution; the completed native chat/warm endpoint repeat below checks temporal variation.

These compare complete serving configurations, not quantization alone: EXL3 uses a separate pinned vLLM image with TP4 plus expert parallelism, packed-cache adapters and bounded calls for one unsupported FlashInfer shape. Native/NVFP4 use the B12X runtime. Exact image/package/settings metadata is in results.json. The EXL3 runtime reports 12,781,811 KV-cache tokens, about 42.61 full 300K contexts before overhead, with 32 scheduler slots. That is a capacity estimate, not a measured 32-user full-window result.

All nine reasoning, 12 cold-context and 12 prefill cells also completed with zero errors, empty answers, output caps and request-counter mismatches. Reasoning aggregate throughput at concurrency 1/8/32 averaged 101.01/330.33/526.01 tokens/s, versus native 131.55/425.19/830.06. Mean leg-median visible-answer latency was 9.87/25.22/58.10 seconds, versus native 9.65/16.76/30.14. Single-user visible latency was similar in these runs, while high-concurrency latency was materially higher.

EXL3 cold first-event latency at concurrency 1/8/32 averaged 8.447/34.714/140.433 seconds, versus native 5.182/24.709/104.618. Prefill at 8K/32K/128K/285K averaged 1.684/6.648/26.366/60.580 seconds, versus native 0.976/3.925/15.606/37.875. Means and observed ranges are in the figures and JSON. Hashing, pre-upload and bounded kernel probes occurred during earlier quality-generation phases and ended before timing; OS/Engram cache history was not standardized. Independent behavior grading and exact portable package qualification passed. The standalone checkpoint, serving recipe and separate rebuilt-image acceptance evidence are published at the revision in released-models.json. No zero-refusal or fastest-format claim is made here.

EXL3 3.5-bit capability update

The larger expert pack passed the same full capability gates: 1,280/1,319 GSM8K, 156/164 HumanEval, and 11,787/14,042 strict MMLU. The original base scored 1,279, 157 and 11,920. The strict MMLU difference is −0.947 percentage points, within the unchanged one-point engineering gate. There were no API errors or empty answers; 733 MMLU answers reached the eight-token cap. The paired outcomes and limits are in exl3-35-full-capability.json.

The smaller expert pack scored 11,859 on strict MMLU, so additional bits did not uniformly improve this single measured run. These are complete runtime-plus-format comparisons with native dense layers and the same attention edit, not a controlled quantization-only study. All six editing tasks passed, and the Claude tool-memory fixture reached 259,121 input tokens with a 300,000-token window, six file reads and no compaction. This is a bounded fixture, not general long-context reliability. Exact package qualification and checkpoint publication are complete; immutable revisions are in released-models.json and independent behavior results are below.

Completed four-format chat and warm-agent comparison

All 36 EXL3 3.5 chat/warm cells completed with zero API errors, empty answers, output caps or request-counter mismatches. Chat aggregate throughput at concurrency 1/32 averaged 129.31/480.84 tokens/s, versus native 146.84/831.66 and lower EXL3 125.86/452.16. Warm-agent throughput was 101.72/398.05 tokens/s, versus native 130.53/740.90. The larger pack does not uniformly beat the smaller one at every concurrency and remains slower than native in these throughput workloads. Native chat/warm and reasoning endpoint repeats are complete below.

Three repeated legs per cell, with observed ranges shown. All numbers compare complete serving paths on shared hardware, not quantization in isolation. Native and NVFP4 use a different runtime from EXL3. Build, pre-upload and CPU diagnostics ended before upper comparative timing; cache history was not standardized. The other eleven plots in this folder cover first-event latency, complete-turn latency, time per output token and draft acceptance. Raw aggregates and runtime metadata are in results.json.

Completed four-format reasoning comparison

All nine larger-EXL3 reasoning cells passed, with zero API errors, empty answers, output caps or request-counter mismatches. At concurrency 1/8/32, aggregate output throughput was 102.38/318.25/522.62 tokens/s, versus native 131.55/425.19/830.06. Counts include reasoning tokens.

Mean leg-median time to a visible answer was 8.97/21.42/56.17 seconds, versus native 9.65/16.76/30.14. The single-request observed ranges overlap; this does not establish a stable single-user latency win. Mean complete-turn latency at concurrency 1/32 was 10.66/64.31 seconds, versus native 10.83/34.53. Natural response lengths differ, so output throughput and time to an answer can rank configurations differently.

All four configurations recorded approximately 98% prefix-cache hit fractions in these warm-prefix cells. Three measured legs per concurrency; shared hardware and observed min–max ranges, not confidence intervals. The native reasoning endpoint repeat is complete below. Cold-context and prefill measurements for the larger pack are complete below; no independent behavior score is implied by timing integrity.

Completed four-format context comparison

All twelve EXL3 3.5 cold-context and twelve prefill cells completed with zero API errors, empty answers, output caps or request-counter mismatches. Its cold first-event latency at concurrency 1/8/32 averaged 8.48/33.63/133.71 seconds, versus native 5.18/24.71/104.62. Cold-context uses two repetitions at each concurrency.

At approximately 8K/32K/128K/285K input tokens, EXL3 3.5 prefill latency averaged 1.741/6.568/25.581/59.705 seconds, versus native 0.976/3.925/15.606/37.875. Prefill uses three repetitions at each length. At the largest input, NVFP4 measured 37.183 seconds and lower EXL3 60.580 seconds. These are observed means on shared hardware; small differences should not be treated as stable wins.

Both EXL3 candidate collections have now completed their full capability, bounded harness and 69-cell timing protocols. Their larger reported KV budgets are capacity estimates, not a measured 32-user test with every user at 300K. Final exact-package qualification and checkpoint releases are complete, including separate rebuilt-image GPU acceptance for the public EXL3 recipe. Native chat/warm and reasoning repeat controls are complete. No raw generated behavior answers are included here.

Native endpoint repeat and temporal variation

The native chat/warm repeat completed all twelve cells at concurrency 1 and 32 with zero API errors, empty answers, output caps or request-counter mismatches. Benchmark-source bytes and prompt receipts match the original run; all other protocol settings match apart from run identifiers and selecting only these two concurrency points. The points in the main comparison plots are shown without lines between unmeasured intermediate concurrencies.

Three legs per point. All original/repeat observed ranges overlap; they are not confidence intervals. NVFP4 warm-agent throughput at concurrency32 averaged 768.23, with range745.32–789.54, overlapping the repeated native range. The few-percent apparent gain therefore is not a demonstrated stable advantage. Natural response lengths and shared hardware also limit causal attribution. See native-final-repeat-main.json for the exact aggregates and source/prompt hashes. The reasoning repeat also passed all six cells, with identical benchmark source, prompts and other protocol settings. At concurrency 1/32, native throughput repeated at 131.69/823.55 tokens/s, versus original 131.55/830.06. Visible-answer latency repeated at 8.64/32.64 seconds, versus 9.65/30.14; complete-turn latency repeated at 10.05/37.44 seconds, versus 10.83/34.53. At concurrency32, the repeated complete-turn range 37.12–38.01 seconds lies above the original 33.53–36.32 seconds, so ranges do not overlap for every metric. Latency can vary materially even when mean throughput is close. See native-final-repeat-thinking.json; the reasoning plots show only the two measured repeat points.

Final independent behavior comparison

All 21 planned grading jobs completed with the pinned HarmBench classifier. It again passed 95/100 balanced human-majority controls: 45/50 negatives and 50/50 positives. This is an imperfect classifier, and these are behavior-success counts, not generic refusal rates. Only final answers are graded. Empty final answers count as nonsuccesses and remain in the denominator; reasoning traces are not used as a fallback.

One HarmBench case exceeded the classifier’s 4096-token context for rank-2, the bounded NVFP4 drafter, NVFP4 target and both EXL3 packs. These cases remain unclassified; they were not truncated or silently removed. The plots use all attempted prompts and show unknown coverage separately. JailbreakBench here uses the HarmBench classifier as an auxiliary rubric, not official JailbreakBench scoring. Thinking generation uses a 6,000-token output budget; output-limit hits and empty finals are disclosed.

The paired method comparison is inconclusive: on 299 common classified HarmBench cases the native attention edit scored 286 and rank-2 scored 285 (10 versus 9 discordant successes; exploratory exact McNemar p=1). Rank-2 scored higher on auxiliary chat/thinking JBB, 94/91 versus 90/88, but exploratory paired p-values were 0.344/0.375. These tests do not prove equivalence or a universal winner; classifier error, correlated cases and multiple comparisons limit interpretation.

The attention edit remains the default for the validated quantization family. Rank-2 has stronger full-capability counts and higher auxiliary behavior counts, but not a demonstrated overall behavior or speed advantage. Its measured reasoning complete-turn median at 32 requests was 45.35 seconds, versus native 34.53 originally and 37.44 in the repeat; single-request ranges and natural output lengths limit broader claims. Native MXFP4 remains the practical speed default. Among these EXL3 packs, the smaller pack has stronger measured capability/behavior counts and more reported KV capacity; the larger pack has somewhat higher chat throughput at some concurrencies. Higher bits did not guarantee better results. None of these measurements supports a zero-refusal claim.

behavior-final.json contains all aggregates, empty/cap counts, source hashes and exploratory paired comparisons. grading-jobs.json and code/evaluation/judge_behavior_batch.py record the exact batch-grading workflow; export_final_behavior.py checks its integrity. The earlier behavior.json is preserved as historical evidence. No generated behavior text is published. Exact package qualification still gates checkpoint release.

Matched reasoning math control

The unmodified base and all compared candidates completed the same 300-item GSM8K subset with thinking enabled, temperature=1, top_p=1, seed 731, concurrency 16 and an output budget of 6,000 tokens. These are sampled subset results, not official full-test leaderboard scores.

No API errors occurred. Keep strict and flexible scoring separate. The earlier native generator has a different code hash, but comparison with its recovered hash-matching source snapshot confirms that only resume-contract checks changed; prompt construction, sampling, response extraction and scoring did not. The original source, source hashes, response hashes and protocol comparison are included in code/evaluation/evaluate_sampling-native-original.py, thinking-math-comparison.json and generator-protocol-comparison.json. This does not imply bitwise deterministic model outputs or a meaningful one-question quality improvement.

Capability and behavior

One temperature-0 chat completion per capability item, with identical prompts and budgets across runs. HumanEval is executed in a disposable, network-disabled container with an execution-completion marker. Strict MMLU uses an eight-token cap and strict A–D formatting. These protocols differ from official leaderboard evaluations.

Rank-2 capability and runtime collection are complete. Its HarmBench classifier covered 299/300 outputs: one response required 4,949 classifier tokens, exceeding its 4,096-token limit. It scored 285 successes, versus 287/300 for the attention-only reference. On the 299 commonly classified cases, the counts were 285 versus 286, with nine rank-2-only successes and ten reference-only successes. This is no demonstrated improvement, not proof of a meaningful inferiority. We retain the reference for the quantized trials while documenting the rank-2 capability tradeoff. The native engine is not bitwise deterministic: two 128-item runs at concurrency 16 matched full text on 106 items. Math scores were 15/16 and 16/16; multiple-choice scores were 76/96 and 78/96. The paired intervals in the JSON are exploratory and do not include run-to-run numerical variation.

The pinned HarmBench classifier passed 95/100 fixed balanced human-majority controls: 45/50 negatives and 50/50 positives. On 200 standard plus 100 contextual direct prompts, it scored behavior success on 30/300 original-base outputs and 287/300 attention-edited outputs, with complete classifier coverage. Generation used a 2,048-token cap; 1 base and 20 edited outputs reached it. These are behavior-success counts, not generic refusal rates or a universal compliance claim. No attack search was performed. Raw requests and generated answers are omitted.

Claude Code completed a real 260,809-input-token session with Read tools and retained early and final facts. This is a bounded functional fixture, not general proof of 300K reasoning quality. Separate native editing checks passed two tasks each in Claude Code, OpenCode and OMP. The subsequently qualified exact packaged Claude wrapper reached 260,205 input tokens; installed OpenCode and OMP aliases reached 281,477 and 255,419, with retained checked facts and no compaction. OpenCode used stdin; OMP used a verifier-only correction for filename:line-range syntax with the original session preserved. native-installed-harness-long-context.json distinguishes packaged and installed-client evidence and its scope. These are not comparative client-latency measurements.

Provenance

See provenance.json for immutable source revisions and patch digests. The native reference uses pqhaz's published attention edit; the alternative uses mrfakename's published rank-2 edit. We did not run the upstream Heretic optimizer or originate these edits.

The native format already uses four-bit MXFP4 expert weights. The NVFP4 target pack is larger because it carries more block scales; its potential benefit is execution speed. storage.json records tensor bytes, including unchanged FP8 Engram tables, rather than GPU allocation or model-wide bits per weight.

Reproduce the native timing protocol

Run on the Linux GPU host, with Docker, NVIDIA Container Toolkit, nvidia-smi, tmux, hf and uv available. Use a sufficiently large data volume. Existing verified weights can be reused. The server launcher never replaces or stops an existing container.

DS_BENCH_WORK=/data/ds41-benchmark
mkdir -p "$DS_BENCH_WORK"
hf download deepseek-ai/DeepSeek-V4.1-Flash-1017 \
  --revision 1bae0e1088993b9b82de08f59de5be5a34e42170 \
  --local-dir "$DS_BENCH_WORK/base" --max-workers 4
hf download pqhaz/DeepSeek-V4.1-Flash-1017-heretic patch/heretic_patch.safetensors \
  --revision 12d58ad0878b5008e80d720251adb969a0069412 \
  --local-dir "$DS_BENCH_WORK/patch"
uv run --no-project --python 3.13.12 python code/apply_native_patch.py \
  "$DS_BENCH_WORK/base" "$DS_BENCH_WORK/patch/patch/heretic_patch.safetensors" \
  "$DS_BENCH_WORK/edited" \
  --sha256 16e63cd3e79c3590855aad340b5b9dc37f27562bf2d0a40f85268915da221ce2
uv run --no-project --python 3.13.12 python code/materialize_links.py "$DS_BENCH_WORK/edited"
bash code/serve-native.sh "$DS_BENCH_WORK/edited" "$DS_BENCH_WORK/cache"

Wait for http://127.0.0.1:8317/health to succeed. Model loading and preparation take minutes. The patch output preserves the original, with copied edited shards and hardlinked read-only unchanged weights. Metadata is copied separately.

TOKENIZERS_PARALLELISM=false uv run --no-project --python 3.13.12 --with tokenizers==0.23.3 \
  python code/make_context.py --tokenizer "$DS_BENCH_WORK/edited/tokenizer.json" \
  --manifest context-source.json --output "$DS_BENCH_WORK/context.txt"
uv run --no-project --python 3.13.12 python code/bench_interactivity.py \
  --container deepseek-local --corpus "$DS_BENCH_WORK/context.txt" \
  --prompt-cache "$DS_BENCH_WORK/prompts" --out "$DS_BENCH_WORK/thinking-results" \
  --arm native-reproduction --load-label YOUR_MEASURED_LOAD_CONDITION \
  --thinking --temperature 1 --top-p 1 --max-output 16384 \
  --workloads agent-warm --concurrency 1,8,32 --requests 4 --repeats 3

The context generator was checked against the measured corpus hash. code/measured/ preserves the exact measured timing sources; code/ adds configurable corpus paths and load labels without changing the request/timing logic. Every run records its own code hash. Keep other API traffic and conversion/build jobs out of measured cells; record any shared GPU load. Do not upload raw telemetry containing host/process identifiers without reviewing it.

Reproduce capability and behavior scoring

code/evaluation/ contains the dataset builders, request generators, executed-code scorer, paired comparator and HarmBench judge. The builders fetch immutable source revisions and check content hashes. They also verify the five derived dataset hashes and the classifier validation set. Raw inputs and outputs stay in a separate local evaluation directory.

DS_EVAL=/data/ds41-evaluation
DS_EVAL_CODE="$PWD/code/evaluation"
DS_EVAL_IMAGE=ghcr.io/local-inference-lab/vllm@sha256:55e477ad62ae15a77c9b869e8fb8e2d958f6edcc4d95e306adfa89dee9ed19df
mkdir -p "$DS_EVAL"
docker run --rm --memory 2g --cpus 1 \
  -e NVIDIA_VISIBLE_DEVICES=void -e CUDA_VISIBLE_DEVICES= \
  -v "$DS_EVAL_CODE:/project:ro" -v "$DS_EVAL:/out" \
  --entrypoint /bin/bash "$DS_EVAL_IMAGE" -lc \
  'python /project/prepare_evaluation.py --out /out/cache/eval && python /project/prepare_overnight.py --source-dir /out/cache/eval --out /out/overnight'

Run the original base and edited model serially, using fresh output directories. Set DS_EVAL_ARM=base for the unmodified model and DS_EVAL_ARM=edited for the patched model, and set DS_EVAL_CONTAINER to the actual running container name. Run the collection block once for each model; only one four-GPU serving container should be running. Long generation commands belong in a detached terminal session. The original base/reference runs used the combined cache/eval/dataset.jsonl; subsequent candidate capability runs use overnight/capability-full.jsonl, with identical per-item capability prompts and budgets. The combined file also contains legacy opening-prefix probes, which are not validated behavior-success results.

DS_EVAL_ARM=edited
DS_EVAL_CONTAINER=deepseek-local
uv run --no-project python code/evaluation/evaluate_full_response.py \
  --dataset "$DS_EVAL/overnight/capability-full.jsonl" \
  --out "${DS_EVAL}/eval/runs/${DS_EVAL_ARM}" --arm "$DS_EVAL_ARM" \
  --root /model --container "$DS_EVAL_CONTAINER"
bash code/evaluation/run_humaneval_scoring.sh \
  "${DS_EVAL}/eval/runs/${DS_EVAL_ARM}" "$DS_EVAL/cache/eval/dataset.jsonl"
uv run --no-project python code/evaluation/evaluate_full_response.py \
  --dataset "$DS_EVAL/overnight/behavior-full.jsonl" \
  --out "${DS_EVAL}/eval/runs/${DS_EVAL_ARM}-behavior" --arm "${DS_EVAL_ARM}-behavior" \
  --root /model --container "$DS_EVAL_CONTAINER"

Wait for each generation's generation.done before its scoring step. After both models have been collected and scored, compare their capability results: uv run --no-project python code/evaluation/compare_quality.py --dataset "$DS_EVAL/overnight/capability-full.jsonl" --reference "$DS_EVAL/eval/runs/base" --candidate "$DS_EVAL/eval/runs/edited" --output "$DS_EVAL/comparison.json" --final. The generators fix temperature 0, seed 731, a neutral helpful-assistant system message and concurrency 16. GSM8K and HumanEval have 1024-token caps; strict MMLU has an eight-token cap. HumanEval uses the pinned CPU image, a read-only/no-network container, per-case 512 MiB address-space and three-second CPU/wall limits, and a completion marker.

For HarmBench, fetch the classifier and its exact official prompt templates. After generation, stop only the serving container created above to free GPU memory for the independent judge; restart it afterward. The example uses GPU 1 on the tested four-GPU layout.

hf download cais/HarmBench-Llama-2-13b-cls \
  --revision bda705349d1144fa618770bea64d99ce54e3835b \
  --local-dir "$DS_EVAL/harmbench-judge"
uv run --no-project python code/evaluation/fetch_judge_prompts.py \
  --output "$DS_EVAL/overnight/harmbench-prompts.json"
mkdir -p "$DS_EVAL/judged"
docker stop "$DS_EVAL_CONTAINER"
docker run --rm --gpus '"device=1"' --network none --memory 64g --cpus 4 \
  -v "$DS_EVAL:/work:ro" -v "$DS_EVAL/judged:/results" \
  -v "$DS_EVAL_CODE:/project:ro" --entrypoint /opt/venv/bin/python \
  "$DS_EVAL_IMAGE" /project/judge_harmbench.py \
  --runs base-behavior edited-behavior --prompts /work/overnight/harmbench-prompts.json --output /results
docker start "$DS_EVAL_CONTAINER"

The judge first validates itself on 100 fixed human-majority controls, then scores the 300 standard/contextual cases. Inputs exceeding its 4096-token context are reported as unclassified, never silently truncated. XSTest and other legacy probes are present in the normalized sources but have no certified refusal scores here. These utilities reproduce the declared protocol; numerical variation and resource-limited code execution can affect repeated scores. record_runtime.py can capture the server configuration and private local weight identities; retain raw responses and private identity files locally.

Results and new benchmark code are MIT-licensed. Upstream models, datasets and runtime components retain their respective licenses. This dataset does not redistribute their model weights or benchmark prompt corpora.

Published checkpoint revisions

• Infatoshi/DeepSeek-V4.1-Flash-1017-MXFP4-uncensored
• Infatoshi/DeepSeek-V4.1-Flash-1017-NVFP4-uncensored
• Infatoshi/DeepSeek-V4.1-Flash-1017-EXL3-2.875bpw-uncensored
• Infatoshi/DeepSeek-V4.1-Flash-1017-EXL3-3.5bpw-uncensored