license: other
license_name: openmdw-1.1
license_link: https://openmdw.ai/license/1-1/
base_model:

• nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16
  base_model_relation: finetune
  quantized_by: AtomicChat
  pipeline_tag: image-text-to-text
  library_name: gguf
  tags:
• atomic-chat
• atomic-dynamic
• nvidia
• nemotron
• nemotron_h_moe
• moe
• gguf
• imatrix
• llama.cpp
• abliteration
• refusal-direction

How to Run NVIDIA Nemotron 3.5 Bolt 74B-A9B Without Refusals Locally

NVIDIA's refusal behaviour projected out of the released weights, baked into Atomic Dynamic GGUF builds and measured against the original.

• On held-out JailbreakBench prompts refusals fall from 97.5% to 0%, on Russian red-teaming prompts from 70% to 0%, and with thinking on from 30 of 30 to none. MMLU stays where it was: 81.35% before, 81.75% after (2000 questions, McNemar p = 0.29). Tool calls and long context are unaffected.
• The refusal is written into the residual stream by every kind of block: the edit covers the Mamba-2 outputs, the attention outputs, the shared experts and the latent projection through which all routed experts write.
• Same Atomic Dynamic layouts and importance matrix as our Nemotron 3.5 Bolt quants; each file is the size of the original build with the same name.

The builds

KLD and top-1 are against the original, unmodified BF16 model on our held-out
neutral text (24 chunks of 4096 tokens), so they hold the edit and the quantization
together. For scale, our plain builds of the original model with the same layouts
are 0.0147, 0.0826 and 0.1941 (on 95 chunks): the edit itself adds about 0.002 to 0.007.

The baked files themselves, thinking off, on the held-out prompts: each of the
three refused none of the 81 English, 100 Russian and 250 XSTest prompts. Degenerate
replies among the Russian ones: 1 on AD-Q4_K_M and AD-IQ2_XS, 4 on AD-IQ1_M_S.
The thinking-on check was run on the BF16 edit (below), not on each quantized file.

For images, use the vision projector from the
original quants
(the edit does not touch it); image input was not tested on these files.

Running it

hf download AtomicChat/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Uncensored-GGUF \
  --include "NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Uncensored-AD-IQ2_XS/*" --local-dir .
llama-server -m NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Uncensored-AD-IQ2_XS/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-Uncensored-AD-IQ2_XS-00001-of-00016.gguf \
  -ngl 99 -c 32768 --jinja

The chat template is inside every file. Upstream llama.cpp with nemotron_h_moe
support runs them; no patch is needed.

How we did it

Nemotron 3.5 Bolt is a hybrid: 40 Mamba-2 blocks, 8 attention blocks and 40
mixture-of-experts blocks whose routed experts live in a narrow latent space. The
usual abliteration recipes are written for plain transformers and miss most of
this model. Here is what we did instead, step by step, with the numbers that
decided each step.

1. Refusal is a direction

A chat model decides to refuse before it writes the first word, and that decision
shows up as one direction in its residual stream (Arditi et al., 2024). Stop the
weights from writing along that direction and the model stops refusing; the rest
of what it writes is barely touched, because one direction out of 4096 is a very
small part of the stream.

2. Finding it

• Prompts. 416 harmful and 416 harmless English prompts (harmful_behaviors,
  harmless_alpaca), 128 and 128 Russian ones (Aya red-teaming, Russian Alpaca),
  each in the model's own chat template, thinking off. Validation and test prompts
  are separate sets and never reach this step.
• Taps. We read the residual stream at the last prompt token after every one
  of the 88 blocks. Mamba-2, attention and MoE blocks name their outputs
  differently inside llama.cpp, so this took a small patch to its graph.
• Direction. At each depth: mean over harmful prompts minus mean over harmless
  ones, English and Russian pooled. Then the component along the harmless mean is
  removed: without that, in our earlier releases the edit made models refuse
  harmless questions that look like harmful ones.
• Depth. How well the direction separates the two sets (AUROC, Cohen's d) and
  how close the English and Russian directions are:

Refusal becomes readable early, but the English and Russian directions only
agree from the middle of the model on. We took block 66, three quarters of the
depth, where the rehearsal model (below) worked best: separation near its peak
and one direction for both languages. Block 50 was the runner-up and lost on
validation (step 4).

3. Where to cut

Four kinds of matrices write into the residual stream: the output projection of
each Mamba-2 block (40), of each attention block (8), the down projection of each
shared expert (40), and the latent up-projection of each MoE block (40). The
routed experts never write to the stream themselves: they work in a 1024-wide
latent space, their outputs are summed there, and one matrix per block lifts the
sum back to 4096. Editing that one matrix covers every routed expert in the block.
That makes 128 matrices; the router and the embeddings are left alone.

All 128 are needed. In a rehearsal on NVIDIA's Nemotron-Labs-3-Puzzle-75B-A9B,
a public model with the same layout, any subset left the refusals in place
(English validation prompts refused; 99% before the edit):

The refusal is written by every kind of block, and whatever is left unedited
writes it back. The fourth row is all that a LoRA adapter can reach in stock
llama.cpp, which applies no adapter to the latent projections. That is why we
ship baked weights rather than an adapter.

4. The edit

For every one of the 128 matrices:

W' = W - s · r rᵀ W

r is the unit direction, so rᵀW' = (1 - s) rᵀW. At s = 1 a matrix can no
longer write along r at all; at s = 1.25 it writes a quarter of its old
contribution with the opposite sign. We screened the scale as a rank-1 adapter on
the BF16 weights (with a llama.cpp patch that applies adapters to the latent
projections):

Scale 1.0 cleared the plain replies but not the refusals the model reaches while
reasoning. 1.25 cleared both; 1.5 did no better and moved the model further from
the original on harmless prompts. A random direction of the same size, on the same
matrices and at the same scale, changes nothing, so the effect comes from the
direction, not from damaging the weights. None of the MMLU changes at block 66
is significant (McNemar p ≥ 0.13).

5. Checks before release

The numbers in this card come from test prompts that played no part in any
choice: 81 JailbreakBench behaviours, 100 Russian red-teaming prompts and the 250
safe XSTest prompts. The thresholds:

• at most 3% refused per language, and XSTest refusals at most 1 point above the
  original's;
• at most 1% empty or looping replies;
• first-token KL on harmless prompts at most 0.10, and KLD on neutral text at most
  0.05 above the original's;
• MMLU on 2000 questions: no drop of more than 1.5 points that McNemar's test
  calls significant;
• with thinking on (30 harmful prompts, 3072 tokens): at most 1 refused;
• 20 tool calls, all valid;
• a needle at 30k tokens, found at all three depths.

The checks ran split across several machines. Their results were merged only when
the model, the code and the direction file had the same fingerprints on all of
them.

6. Baking and quantizing

The edit is applied to the BF16 weights in f32 and rounded to BF16 once. After
rounding, every one of the 128 matrices keeps a quarter of its write along r
(the ideal 0.25) within 0.0007. The baked BF16 is then quantized with the same
importance matrix and the same Atomic Dynamic layouts as our
original quants,
and measured against the unedited BF16 (the table at the top).

What changed, and what did not

Original against this edit on the released weights (BF16, the edit applied as an
adapter), on the test prompts of step 5.

Validation prompts (104 harmful, 104 harmless per language), replies of up to 64
tokens, the check of the recipe on the released weights: none of the English or
Russian harmful prompts refused.

Refusal is counted by the opening of the reply, so it is indicative, not a
judge.

Limitations

• The scale of 1.25 moves the model a little further from the original on harmless
  prompts than 1.0 would (first-token KL 0.062 against 0.044); 1.0 did not clear
  the refusals the model reaches while thinking.
• The refusal count reads the opening of each reply; a soft refusal phrased as
  an answer would be missed.
• MMLU and the thinking check use fixed question sets; they show a large change,
  not a small one.
• We publish the baked files only. The same edit as a LoRA adapter needs a
  llama.cpp patch on this architecture (upstream applies no LoRA to the latent
  projections), so it is not offered here.

Responsible use

This model has its learned refusal behaviour removed, so it will answer
requests the original declines. Nothing about it makes the output safe, correct
or lawful, and any deployment needs its own access controls and policy
enforcement.

License and credits

OpenMDW-1.1, as the original. Base model Nemotron 3.5 Bolt 74B-A9B by
NVIDIA. Direction, edit, quantization and
evaluation by AtomicChat. The method follows Arditi et al., Refusal in
Language Models Is Mediated by a Single Direction (2024).