license: other
license_name: nvidia-open-model-license
license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/
base_model: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16
library_name: transformers
pipeline_tag: text-generation
tags:
- nemotron
- nemotron-h
- abliterated
- obliteratus
- mixture-of-experts
- hybrid-mamba-transformer
- research-use
Nemotron-3-Super-120B-A12B-BF16-Abliterated-Conservative
Abliterated derivative of nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16,
produced via the OBLITERATUS pipeline (variant: conservative).
Research use only. This model has had its refusal-direction subspace
excised. It will comply with harmful prompts that the base model would
have refused. It is intended exclusively for safety, alignment, and
red-team research. Do not deploy publicly. Honor the upstream NVIDIA
Open Model License when redistributing.
What was changed
OBLITERATUS removes the refusal direction(s) from a model's residual
stream by:
- Profiling activations on harmful vs harmless prompt pairs.
- Identifying the principal refusal direction(s) via a generalized
eigenvalue / SVD pass on contrastive activation differences. - Projecting the identified direction(s) out of attention/MLP/MoE expert
weights ("excise") at every block where they contribute meaningful
signal.
The conservative variant uses the configuration inconfigs/obliteratus/nemotron3_120b_conservative.yaml:
n_directions=2, EGA + jailbreak + head-surgery + SAE all enabled
See docs/OBLITERATUS.md
for the full pipeline (8 architecture-specific patches, the lm_head
shard-load fix that unblocked Nemotron-3 120B specifically, and the
external eval suite definitions).
Verification metrics
In-process OBLITERATUS verification at the end of the abliteration run:
| Metric | Value |
|---|---|
| Load-time perplexity | 18.85 |
| Post-EXCISE perplexity | 23.26 |
| Reference-text perplexity | 3.9 |
| Coherence (10 fixed prompts) | 100% |
| Internal refusal rate (30 prompts, T1–T7 tiers) | 13% |
| First-token KL vs base (100 harmless prompts) | 1.27 |
External benchmarks (TalonLab abliteration_lite suite)
- AdvBench compliance: 99.62%, 99.23%, 99.81% across 3 independent runs (520 prompts each, stochastic sampling at temp=0.7). Pre-abliteration baseline ≈ 0%.
- Tier cleanup: T3/T5/T7 (high-harm tiers) reduced to 0% refusal.
IFEval (Google instruction-following benchmark, 541 prompts, zero-shot,
chat-template applied, run 2026-05-01 on 8× H200):
| Metric | Score |
|---|---|
prompt_level_strict_acc (headline) |
30.13% |
inst_level_strict_acc |
44.00% |
prompt_level_loose_acc |
38.63% |
inst_level_loose_acc |
51.32% |
For comparison, frontier base models typically score ~75–85% onprompt_level_strict_acc. The drop here suggests the conservative
abliteration (n_directions=2, EGA + jailbreak + head-surgery + SAE all
on) damaged instruction-following capability noticeably — a known
trade-off when the projected directions are not strictly orthogonal to
the instruction-following subspace.
RULER 32k has not yet been run on this checkpoint. The TalonLab eval
suite is configured for it but the run is pending.
Architecture notes
- Hybrid Mamba + attention + sparse MoE (Nemotron-H family), 88 layers.
- 512 experts × 22 active per token at the MoE layers.
- BF16 weights, ~240 GB on disk.
lm_head.weightis a separately-trained, untied parameter (the 120B
variant has_tied_weights_keysremoved). Loading viafrom_pretrained(..., device_map="auto")requires the_materialize_lm_head_from_shardspatch in OBLITERATUS — without it,lm_head.weightis left on the meta device and PPL collapses to ~10⁵.
Inference
The model serves cleanly via SGLang (≥ 0.5.10 has native Nemotron-H
support). Use BF16 + tensor-parallel ≥ 4 on H100/H200 class hardware.
Tested deployment: 8× H200 with SGLang --dtype auto --trust-remote-code --chunked-prefill-size 8192 --disable-cuda-graph --disable-piecewise-cuda-graph.
python -m sglang.launch_server \
--model-path surfingpapi/Nemotron-3-Super-120B-A12B-BF16-Abliterated-Conservative \
--tp 8 \
--trust-remote-code \
--dtype auto
Citation / provenance
If you use this model for research, please cite:
- Base model: NVIDIA Nemotron-3 Super 120B A12B BF16
(https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16) - Abliteration pipeline: TalonLab OBLITERATUS
(https://github.com/MindPointGroup/talonlab-modeltraining/tree/main/src/talonlab/obliteration) - Abliteration technique (original): Arditi et al., Refusal in Language
Models is Mediated by a Single Direction (2024).