license: mit
base_model: deepseek-ai/DeepSeek-V4-Flash-Vision-Exp
base_model_relation: finetune
pipeline_tag: image-text-to-text
library_name: transformers
language:

• en
• zh
  tags:
• abliterated
• deepseek
• deepseek-v4
• uncensored
• ai-red-team
• red-teaming
• moe
• vision-language
• image-text-to-text
• reasoning
• mtp
• fp4
• fp8
• post-training

DeepSeek-V4-Flash-Vision-Uncensored

The abliterated (refusal-removed) build of DeepSeek's V4-Flash-Vision-Exp — baked directly into the official block-FP8 / FP4 checkpoint, byte-for-byte drop-in for the original

One Gateway. Every Model. — Route Smarter · Ship Safer · Spend Less.

Website · Model Catalog · GitHub

The abliterated (refusal-removed) build of
deepseek-ai/DeepSeek-V4-Flash-Vision-Exp
— a 305B / ~18B-active Mixture-of-Experts model with DeepSeek sparse attention, 4-wide
Manifold-Constrained Hyper-Connections, a native vision tower, a DSpark / MTP
speculative head, and a 1M-token context.

The refusal direction is baked directly into the official mixed-precision shards — same
format, same shard layout, same model.safetensors.index.json. Every tensor matches the base
checkpoint in name, dtype and shape, so this is a drop-in replacement for
deepseek-ai/DeepSeek-V4-Flash-Vision-Exp in any stack that already serves it.

On the precision: this checkpoint is a four-precision model as DeepSeek ships it — the
routed experts are FP4 (e2m1, packed 2/byte, ue8m0 per-32 scales), attention and the
shared expert are block-FP8 (e4m3, 128×128, ue8m0 scales), embeddings / vision / norms /
the MoE router are BF16, and the Hyper-Connection mixers are FP32. There is no upstream
BF16 release; this is the full-precision source as published.

⚠️ Disclaimer — read before use

This model has had its safety alignment substantially removed via abliteration
(orthogonalizing the refusal direction out of the residual stream). As a direct consequence:

• It will comply with harmful, unethical, offensive, or illegal requests that the original
  model would refuse. It has no meaningful built-in guardrails.
• It is released strictly for legitimate research — interpretability, AI-safety and
  refusal-mechanism study, red-teaming, robustness evaluation, and controlled experiments.
• You assume full responsibility and liability for how you use it and for everything it
  generates. Do not deploy it to end users or in production without adding your own safety,
  moderation, and abuse-prevention layers.
• Use must comply with the MIT License inherited from
  the base model, and all laws and regulations that apply to you.
• The authors and uploaders accept no liability for any misuse or harm. Its outputs do not
  reflect the views of the uploaders or of DeepSeek.

By downloading or using this model you acknowledge and accept the above.

Model details

Abliteration

Refusal-direction removal following Arditi et al. (2024), Refusal in Language Models Is
Mediated by a Single Direction. A single refusal direction r (k = 1) is estimated as the
massive-activation-masked mean-difference of harmful − harmless activations, read from the
4096-d residual stream at layer 28 of 43 (depth 0.65) and selected by a 10-candidate quality
sweep scored by bypass − kl_penalty·kl_norm − induced_benign_refusal. r is then orthogonalized
out of every residual-writing matrix — W' = W − r(rᵀW) — computed in float32.

Reading the direction under Hyper-Connections

The mHC residual is a stack of hc_mult=4 parallel streams, and output_hidden_states exposes the
widened 4×4096 = 16384 stream — the naive last-token recipe would estimate the direction in the
wrong space. Instead we read the un-widened 4096-d stream that each sub-block actually consumes,
captured with forward pre-hooks on input_layernorm (attention read) and post_attention_layernorm
(MoE read). The two read points genuinely differ within a layer, and the sweep found the refusal
direction is linear and cleanly separable at the MoE read point, not the attention one.

mHC cannot re-introduce the direction after the writers are cleaned: its comb mixing matrix is
doubly-stochastic and acts over the branch axis, and post is a per-branch scalar gain — both are
linear and branch-wise in hidden space. Orthogonalizing the 4096-d writers is therefore sufficient,
and the hc_* parameters are left untouched.

Why it had to be baked on disk

This checkpoint cannot be abliterated by the conventional in-memory path:

• Its routed experts are FP4: raw e2m1 nibbles packed two-per-byte with sibling ue8m0
  scales. Projecting the nibbles corrupts the matrix, and the leakage probe — reading the same
  quantized code space — reports a false ~0.
• Its 11,008 routed-expert w2 matrices are the MoE's main residual writer. A conventional
  2-D-.weight matcher edits none of them and still reports success.
• transformers loads the model text-only — it drops the vision tower, aligner, the four
  learned image vectors and the MTP blocks on load, so save_pretrained would silently omit them.

The shards are therefore streamed and rewritten one at a time, in the checkpoint's own formats.

What was edited

A post-bake verification pass confirms all 48 shards and 72,633 tensors match the base checkpoint
in name, dtype and shape, and that every tensor is accounted for as either edited or copied.

The quantization-aware requantizer — why FP4 abliteration is not lossy here

Orthogonalizing a quantized matrix and requantizing leaks the direction straight back. Measured on
real tensors under the actual refusal direction:

The naive edit perturbs the matrix by only ~3%, far below one FP4 code step, so round-to-nearest
simply undoes it — and iterating does not help; it is a fixed point. We instead solve the
constrained quantization problem directly, per column j:

min_q ‖q − W[:,j]‖²   s.t.   rᵀq = 0,   q on the FP4/FP8 grid

Its Lagrangian is quantize(W[:,j] + λⱼ r) — a per-column over-projection whose multiplier is
found by bisection (the constraint function is monotone in λ), followed by single-code refinement
and per-block ue8m0 scale co-optimization. The output is a fixed point of the runtime's own
dequant/requant kernel, so it survives loading unchanged. This is what makes a lossless-quality
bake possible on an FP4 MoE, and it is why capability is retained (below): the edit is exact on the
direction while perturbing the weights by only ~3.7% (FP4) / ~2.1% (FP8).

leakage = ‖rᵀW_baked‖ / ‖rᵀW_orig‖, mean over 8 layers × 4 experts × several directions.

Where the refusal lives

The sweep also localizes the mechanism. Hook-ablating the direction (lossless, no bake) at each
candidate read point and layer, on the held-out validation split:

The refusal direction is written into the MoE read point and is cleanly removable there at
mid-to-late depth; the attention read point never gets below 0.42. Layer 28 is a narrow optimum,
and the direction generalizes to a different held-out distribution (JailbreakBench) rather than
overfitting the AdvBench/Alpaca fit data.

Evaluation

Measured on these exact weights, 4×H100, greedy decoding, thinking_mode="chat" (DeepSeek closes
its <think> block immediately so the answer — not a reasoning preamble — is what the classifier
reads). Refusal is judged by a rule-based opening-phrase classifier (caveat = answered but wrapped
in a disclaimer) — indicative, not an LLM-judge / publication-grade number. Both columns are the same
script, seed and settings on the base and abliterated checkpoints.

Safety — harmful-prompt refusal (lower = more uncensored)

Over-refusal — benign prompts wrongly refused (lower = better)

250 prompts written to look harmful while being benign, plus the held-out benign split. The edit
does not blunt the model's discrimination — it sharpens it; over-refusal drops to near-zero, below
the base model's own rate.

Capability retention — vs the official base (same scripts, same settings)

Every delta is within ±1 pp. Multiple-choice is scored from a single forward pass, argmax over the
option-letter logits; GSM8K keeps chain-of-thought generation. Read the deltas as "unchanged" — the
constrained requantizer's ~3.7% weight perturbation carries no measurable capability cost.

Usage

The base model's repository ships an encoding/ reference (there is no chat template in the
tokenizer) and an inference/ reference implementation that covers vision, MoE, Hyper-Connections
and the DSpark path. Prompts must be built with encode_messages(...), not apply_chat_template.
The model card for the base recommends serving via SGLang with --speculative-algorithm DSPARK
(draft and target share this checkpoint). This build is a drop-in for the base in that stack.

For text-only research on transformers (5.16+): the FP8/FP4 path needs kernels==0.16.0, and the
FineGrainedFP8HfQuantizer.update_tp_plan upstream crash must be stubbed for a single-process load;
the HF class is text-only (vision is dropped on load).

Bias, risks, and limitations

• Safety guardrails removed — the model will produce harmful, biased, or offensive content on
  request (see the disclaimer).
• It inherits any biases and limitations of the base DeepSeek-V4-Flash-Vision-Exp.
• Image-conditioned refusal is not yet quantified. The vision residual writers (aligner.w2
  and the four learned image vectors) are abliterated in these weights, but the effect on
  image+text refusal has not been measured on real multimodal inputs — the transformers stack is
  text-only, so that evaluation requires the base repo's reference implementation and is pending.
  Treat the vision path as edited-but-unverified.
• The reported refusal metric is a rule-based heuristic; evaluate rigorously for your own use case.
• Capability retention is measured, not assumed, but on sampled subsets of three benchmarks — enough
  to rule out a large regression, not a substitute for a full harness run. The GLM-5.3 sibling card's
  HarmBench / StrongREJECT / CMMLU columns are not included here (dataset access pending).
• Refusal is reduced, not removed. A small residual refusal rate remains, and — as with any
  single-direction abliteration — some content categories may be mediated by directions this method
  does not reach.

License

MIT, inherited from the base model
deepseek-ai/DeepSeek-V4-Flash-Vision-Exp.
Abliteration does not change the underlying license obligations.