license: mit
library_name: transformers
pipeline_tag: text-generation
base_model:

• deepseek-ai/DeepSeek-V4-Flash-0731
• apetersson/DeepSeek-V4-Flash-0731-Abliterated-FP8
  base_model_relation: quantized
  tags:
• deepseek-v4
• mixture-of-experts
• abliterated
• nvfp4
• fp8
• model-surgery
• dspark
• modelopt
• vllm

DeepSeek-V4-Flash-0731-Abliterated-NVFP4

This is an ABLITERATED derivative — its refusal behaviour has been altered by
rank-1 direction projection. It is not the stock DeepSeek-V4-Flash-0731 and it is
not nvidia/DeepSeek-V4-Flash-NVFP4. Deploy accordingly.

The point of this build: the speculative drafter is abliterated too, and it is
in the box.

DeepSeek V4's three-stage DSpark drafter ships inside this checkpoint (NVIDIA's
recipe lists mtp.* under ignore, so the quantizer skips it rather than
dropping it). The source checkpoint had already applied the refusal-direction
projection to all three drafter stages as well as the main model, and because
mtp.* is passed through untouched, that surgery survives here bit-for-bit.

Why that combination matters: a speculative drafter earns its keep through
acceptance rate. Pair an abliterated target with a stock drafter and the
drafter keeps proposing refusal-shaped continuations that the target no longer
wants — proposals that get rejected, which is exactly the throughput you were
speculating to gain. Aligning the two is the reason this artifact exists.

Routed MoE experts are NVFP4; attention, shared experts, LM head and the
DSpark/MTP drafter stay in the source formats (FP8 / native MXFP4 packing) — the
same mixed-precision layout NVIDIA ships.

• 163.49 GiB (48 shards, 175,550,788,904 bytes)
• mtp.* preserved: 4,705 keys, not quantized — the DSpark drafter is intact
• Source format note: the input checkpoint is published as "FP8", but its routed
  experts are in fact native MXFP4 with FP8 attention. That is why the
  conversion is --cast_mxfp4_to_nvfp4 (a repack of the experts) with attention
  and MTP passed through — which is also why the abliteration surgery on the
  residual writers survives bit-for-bit on the passed-through tensors.

Verified on the bake host (MEASURED)

Loaded and generated on vLLM 0.26.0, tensor-parallel 8 × RTX PRO 2000
Blackwell (16 GB each) plus CPU offload — i.e. it runs on a workstation
without a 12-card frame, at the cost of speed:

--tensor-parallel-size 8 --cpu-offload-gb 12 --kv-cache-dtype fp8
--gpu-memory-utilization 0.88 --trust-remote-code --enforce-eager
--max-model-len 512   # smoke-test value, raise for real use

NOT measured on this artifact: throughput (t/s), DSpark/MTP acceptance rate,
benchmark scores, long-context behaviour, and any systematic evaluation of how
far the abliteration generalises. Those numbers are absent because they were not
run — not because they were disappointing. The smoke test above used
--max-model-len 512 and --enforce-eager, so it says "it loads and speaks
coherently" and nothing about speed.

Serving notes

• vLLM ≥ 0.26 verified here. Older builds (e.g. 0.21) do not read this
  mixed-precision NVFP4 layout.
• SGLang is the stack NVIDIA's sibling build targets, with DP-attention,
  expert parallelism and MTP; not exercised here.
• On this no-NVLink box, NCCL_P2P_DISABLE=1 and
  --disable-custom-all-reduce were required for TP=8.
• Twelve 16 GB cards (191 GiB) hold the model with ~28 GiB of headroom and no
  CPU offload; eight cards need roughly 12 GiB offloaded per the config above.

DSpark (MTP) — it is inside this checkpoint

DeepSeek V4's speculative drafter ships within the checkpoint rather than as
a companion file, because NVIDIA's NVFP4 recipe puts mtp.* in the ignore
list — so the drafter is skipped by the quantizer, not dropped by it, and
survives at its source precision:

"ignore": ["*.attn.*", "*.ffn.shared_experts.*", "head", "mtp.*"],
"moe_quant_algo": "NVFP4", "group_size": 16,
"producer": {"name": "modelopt", "version": "dsv4-nvfp4-experts"}

Verified present in this build: 4,705 mtp.* keys, zero NVFP4 scale siblings
under mtp.* — i.e. the drafter was passed through untouched.

Provenance for the drafter surgery: upstream ABLITERATION_MANIFEST.json records
edit_mtp: true with 36 edited tensors, the attention output projection in
each of the three DSpark stages alongside main-model layers 10-42, rank-1
projection at lambda 3.5. Its metadata
describes the real three-stage DSpark used by 0731:

Two practical consequences, both learned the hard way on this box:

1. You do not need a separate support GGUF. The GGUF route requires loading a
   standalone DSpark support file alongside the target model; here the drafter is
   already resident. (For the record, on the GGUF side we measured that
   --ssd-streaming and --mtp are mutually exclusive in ds4 — DSpark there
   demands full residency, which a single 16 GB card cannot give an 80 GiB model.
   The NVFP4/vLLM route has no such constraint because it is full-residency by
   design.)
2. Keeping the drafter unquantized is deliberate. A speculative drafter earns
   its keep through acceptance rate; quantizing it to 4 bits to save ~20 GiB
   would trade the very accuracy that makes speculation pay. We kept NVIDIA's
   ignore list one-to-one rather than second-guessing it.

Serving DSpark today: MEASURED status (2026-08-02)

Both major stacks recognise this drafter. Neither could run it on this
host. Reporting both walls precisely, because each costs an evening to rediscover.

vLLM 0.26.0 — understands it, runs out of VRAM.
The correct flag is method: "dspark", not "mtp" (vLLM merged native DSpark
support on 2026-07-01):

--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'

With "mtp" it resolves DeepSeekV4MTPModel then dies on
KeyError: 'model.layers.43.mtp_block.main_norm.weight' — a single-stage MTP path
cannot express a three-stage drafter. With "dspark" that KeyError is gone and
vLLM validates the block shape against the checkpoint metadata: passing
num_speculative_tokens: 3 is rejected with "DSpark requires
num_speculative_tokens >= dspark_block_size (5)". Stack and weights agree.
It then OOMs on every configuration tried — cpu-offload-gb 12/20/24/40,
gpu-memory-utilization to 0.96, max-model-len down to 512 — always with
~23 MiB free per card, which suggests the draft model requires GPU residency
rather than joining CPU offload. Target-only fits; target-plus-drafter does not.
Note also that vLLM raises NotImplementedError: Pipeline parallelism is not supported for this model, and tensor-parallel size must be a power of two — so
on this architecture vLLM caps at 8 cards regardless of how many are installed.
Adding cards does not lift this wall; a stack that shards it differently would.

SGLang (nightly-dev-cu13-20260802) — understands it, hits a quantization-layout bug.
It resolves the architecture as DeepseekV4ForCausalLMDSpark and ships dedicated
DSpark options (--speculative-algorithm DSPARK,
--speculative-dspark-block-size, SPS-table and confidence-STS paths), i.e. a
first-class implementation. On this checkpoint it fails during weight binding:

ValueError: functional_call got multiple values for keys
  ['mlp.experts.w13_blockscale_swizzled', 'mlp.experts.w13_weight_scale'],
  which are tied
  (torch/nn/utils/stateless.py, _untie_named_tensors_map)

Same with --disable-cuda-graph, so it is not graph capture. The NVFP4 swizzled
block-scale tensors are tied to their weight-scale siblings in a way this path
does not expect. Run with --tp 8 --ep-size 8; note that including the display
GPU causes an unrelated OOM, so pin the device list to idle cards.

Why more cards do not (yet) help. vLLM's DeepseekV4ForCausalLM does declare
SupportsPP, but the draft model DeepSeekV4MTP does not — and SGLang states the
same constraint outright: "Currently DSpark speculative decoding only supports
pp_size == 1." With pipeline parallelism unavailable to the drafter and
tensor-parallel size restricted to powers of two, DSpark is capped at 8 cards on
this architecture no matter how many are installed. Target-only can use
pipeline parallelism (we loaded it across 10 cards, 135 GiB resident), though on
this no-NVLink host that configuration then hung in shared-memory broadcast — a
separate problem, and not one this checkpoint causes.

So: the drafter is present, abliterated, and understood by both stacks — and
unexercised here. Acceptance rate and speedup are unmeasured for those two
reasons and no other. If you clear either wall, numbers are very welcome in the
discussions tab.

Acceptance rate and the resulting speedup on this artifact are NOT measured.
Speculative decoding is famously environment-dependent — on this same lab we have
measured it losing on CPU MoE inference (batch verification wakes proportionally
more experts: 0.658–0.743× on Kimi-K3 GGUF), while GPU reports for DeepSeek V4
class models show clear wins. Measure it on your own box before assuming either.
To exercise it you need a stack with DeepSeek MTP support enabled (vLLM ≥ 0.25 or
SGLang); our verification above ran the target model only.

Provenance (revisions)

See also ABLITERATION_MANIFEST.json (inherited from the FP8 abliterated
source) and the staged SHA256SUMS for every file in this tree.

Quantization recipe (ignore list 1:1)

Producer stamp written into config.json / hf_quant_config.json:

{ "name": "modelopt", "version": "dsv4-nvfp4-experts" }

Hardcoded exclude / HF ignore list (public recipe, not modified):

*.attn.*
*.ffn.shared_experts.*
head
mtp.*

Meaning:

• Converted: routed MoE experts only → NVFP4 (moe_quant_algo: NVFP4,
  group size 16).
• Not converted (passthrough): attention, shared experts, LM head, and all
  mtp.* (DSpark / multi-token prediction) tensors, remaining in the source
  mixed formats.

What was actually run (MEASURED bake path)

Command shape (see bake log; device was CUDA for the cast export on this host):

python3 quantize_to_nvfp4.py \
  --amax_path /path/to/amax-nvfp4-experts-abl \
  --source_ckpt /path/to/0731-abliterated-fp8 \
  --output_ckpt /path/to/0731-abliterated-nvfp4 \
  --device cuda \
  --cast_mxfp4_to_nvfp4 \
  --overwrite

Activation PTQ (ptq.py / multi-GPU torchrun+NCCL calibration) was NOT run.
On the 10×16 GB bake host the full PTQ load path does not fit; input_scale
values were stamped from a synthetic constant amax:

• INPUT_AMAX = 6.0 for every routed expert projection
  → input_scale = 1/448
• Weight path under --cast_mxfp4_to_nvfp4 uses the lossless MXFP4→NVFP4 cast
  (weight amax not used for the casted weights).

Synthetic amax dump: 66 048 keys (33 024 input + 33 024 weight),
SHA-256 bcfcd9e53ef3bcf714b0f943448b8f6163c88f9c380297875985293089ecc6b2.

Cast statistics (MEASURED from export log)

Abliteration (inherited; not re-done here)

The NVFP4 cast does not re-apply abliteration. Surgery remains exactly as in
the FP8 abliterated source:

• Method: refusal-direction projection from attention residual writers.
• Rank 1, λ = 3.5, layers 10–42 inclusive, plus corresponding MTP attention
  wo_b writers (36 tensors total).
• Three fixed-point FP8 requantization iterations at abliteration time.
• Direction SHA-256:
  6e4d8a8f3aa9e21795faab2c5b14d29b019acdf2ddbfbd8238430458a5837fe0.

Because attention and MTP are on the ignore / passthrough list, those edited
tensors are copied, not requantized, by this bake.

Format and use

Architecture, tokenizer, official message encoding, 1 M context declaration,
and attached DSpark tensors come from the 0731 line. Use the included
encoding/ directory and the upstream model card for prompt formatting.

Recommended upstream sampling defaults: temperature=1.0, top_p=0.95 for
agentic scenarios, top_p=1.0 otherwise.

Serving (NOT verified for this artifact on the bake host)

NVIDIA’s NVFP4 Flash card documents:

• SGLang, or
• vLLM ≥ 0.22 (and ≥ 0.25 for DSpark features per project notes).

The bake host’s stock vLLM 0.21.0 is not the verified stack for this
checkpoint. Do not treat local 0.21 install as a green light.

Reference NVIDIA deploy notes (for the official NVFP4 sibling, not this
abliterated tree): SGLang with TP; vLLM with --kv-cache-dtype fp8 on
Blackwell-class hardware.

Validation status

Structural validation (MEASURED on this tree)

Live re-check of the n3 structural gates against the written files:

Pilot (n3a) also showed CPU vs GPU cast byte-identical on a miniature
source and full attention/MTP passthrough integrity.

Behavioral validation of this NVFP4 artifact

NOT MEASURED. No refusal gauntlet, no StrongREJECT judge, no capability
suite, and no production serve was run against
/mnt/stripe/models/0731-abliterated-nvfp4.

Inherited / proxy evidence (about the source abliteration line only)

The FP8 abliterated source card documents an MLX mixed 2/3-bit deployment proxy
benchmark (oMLX, 2026-08-01). That evidence:

• applies to a quantized MLX derivative of the abliterated line,
• is not a direct measurement of native FP8,
• is not a measurement of this NVFP4 tree,
• must not be copied as scores for this release without re-running.

MEASURED vs ESTIMATE (explicit)

MEASURED

• Source abliterated tree downloaded at HF commit
  7d02640c72a2c8127f116d3d1933ddfec5e4c0fa.
• Bake used public quantize_to_nvfp4.py with --cast_mxfp4_to_nvfp4 and
  producer dsv4-nvfp4-experts.
• Ignore list is exactly
  *.attn.*, *.ffn.shared_experts.*, head, mtp.*.
• MXFP4→NVFP4 weight cast lossless block rate 100%.
• Structural gates above all PASS; disk shard total 163.49 GiB.
• Attention / shared / head / MTP tensors are not NVFP4-rewritten
  (abliteration residual writers preserved by passthrough).
• Full per-file SHA-256 inventory of this tree (see SHA256SUMS).

ESTIMATE / synthetic / not measured

• Activation scales: synthetic INPUT_AMAX=6.0 → input_scale=1/448.
  Not calibrated on real traffic. Quality impact of synthetic input scales
  is unknown.
• Serving stack fit and tokens/s: not measured on this artifact.
  Plan-level “44 t/s with MTP on 12×16 GB” remains an ESTIMATE / goal, not
  a result.
• Refusal / uncensor behavior after NVFP4 cast: not measured.
• Capability preservation (coding, reasoning, long context): not measured.
• Parity with nvidia/DeepSeek-V4-Flash-NVFP4 accuracy tables: not claimed;
  NVIDIA calibrated activations; this bake did not.

Limitations

• Abliteration can affect capabilities beyond refusals.
• Synthetic activation scales may interact poorly with some runtimes or
  sequences even when weights cast losslessly.
• DSpark/MTP behavior under NVFP4+passthrough has not been load-tested here.
• Use access controls appropriate to an abliterated model.

License and attribution

• Upstream DeepSeek weights: MIT (LICENSE).
• Refusal direction: drowzeys/keys MIT notice in NOTICE.
• Quantization recipe: NVIDIA Model-Optimizer (public examples); this card is
  not an NVIDIA release.
• Please credit DeepSeek-AI, the abliteration source maintainers, and NVIDIA
  Model-Optimizer when redistributing or publishing results.

Integrity

Publish only after Ken’s explicit approval. Irreversible Hub upload was not
performed by the n4 staging job. Verify against SHA256SUMS after any copy.