license: other
license_name: openmdw-1.1
license_link: LICENSE
base_model: IstroSec/ThinkingCap-Nemotron-3.5-Lightning-30B-A3B-abliterated
base_model_relation: quantized
library_name: transformers
pipeline_tag: text-generation
language:

• en
  tags:
• nemotron_h
• nemotron-3.5
• thinkingcap
• abliterated
• uncensored
• moe
• mamba
• nvfp4
• nvfp4a16
• fp4
• gptq
• compressed-tensors
• llmcompressor
• vllm
• mtp
  datasets:
• HuggingFaceH4/ultrachat_200k

ThinkingCap-Nemotron-3.5-Lightning-30B-A3B-abliterated-NVFP4A16

A 4-bit weight-only quantization (NVFP4 weights, bf16 activations) of ThinkingCap-Nemotron-3.5-Lightning-30B-A3B-abliterated, the uncensored variant of bottlecapai/ThinkingCap-NVIDIA-Nemotron-3.5-Lightning-30B-A3B.

22.1 GB, down from 65.8 GB in bf16, at KL 0.0148 against the bf16 model. The weights alone take 22 GB, so a 24 GB GPU leaves little room for context; 32 GB and up is comfortable. Activations stay bf16, so the GPU does not compute in FP4; vLLM runs it with Marlin FP4 kernels on Ampere and newer (sm80+). Only a DGX Spark was tested.

Which build? FP8-DYNAMIC (35.0 GB, KL 0.0134) diverges slightly less and needs an Ada or newer GPU; W4A16 (20.5 GB, KL 0.0215) is 1.6 GB smaller and diverges more. Speed was not measured.

Usage

vLLM

Tested 2026-10-23 with vLLM 0.30.0 (vllm/vllm-openai:v0.30.0) on a DGX Spark (GB10), --max-model-len 32768 --max-num-seqs 8, MTP on (5 speculative tokens). Four requests at temperature 0 (a short explanation, a code task, a tool call, one request with thinking off) gave correct answers and a well-formed get_weather tool call. vLLM used its Marlin FP4 weight-only path.

vllm serve IstroSec/ThinkingCap-Nemotron-3.5-Lightning-30B-A3B-abliterated-NVFP4A16 --async-scheduling \
  --reasoning-parser nemotron_v3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}'

Sampling (NVIDIA's recommendation for the base model, which ThinkingCap uses unchanged): temperature 1.0, top_p 0.95. Thinking is on by default; turn it off per request with chat_template_kwargs: {"enable_thinking": false}.

MTP. The head is bf16 and listed in quantization_config.ignore. In the test above (5 speculative tokens, temperature 0, 4 short requests) vLLM accepted 63% of drafted tokens (871 of 1,385, mean acceptance length 4.14); figures from sampling-based runs are not directly comparable.

Memory. On GPUs with less memory, vLLM can stop at startup with max_num_seqs (1024) exceeds available Mamba cache blocks; lower --max-num-seqs.

Transformers

Loads through llm-compressor's loader, with the weights dequantized to bf16 (about 66 GB of memory):

from transformers import AutoModelForCausalLM, CompressedTensorsConfig
from llmcompressor.utils import load_context
with load_context(AutoModelForCausalLM):
    m = AutoModelForCausalLM.from_pretrained("IstroSec/ThinkingCap-Nemotron-3.5-Lightning-30B-A3B-abliterated-NVFP4A16", device_map="cuda",
                                             quantization_config=CompressedTensorsConfig(dequantize=True))

llama.cpp

Not for llama.cpp: this is compressed-tensors format. Use the GGUF builds.

Evaluation

Heretic's evaluation protocol, 100 harmful and 100 harmless prompts, against the bf16 abliterated checkpoint, with thinking off:

Keeping the Mamba projections in FP8 and attention in bf16 cut the all-FP4 build's KL to about a third for 0.6 GB more; GPTQ then cut it by another fifth.

The two refusal columns score the same responses with Heretic 1.2.0's broad marker list and with the abliteration's own marker list (34 refusal phrases such as "i cannot" or "sorry, but i", from its published config.toml). Decoding is greedy, but the batched GPU kernels are not bit-deterministic here: two runs of the bf16 weights gave 21 and 23 refusals. Differences of a few refusals between builds are not significant.

Released Heretic (up to 1.4.0, and its master branch as of 2026-10-23) cannot load this architecture: it finds no abliterable modules under Nemotron's mixer blocks. The abliteration used here was made with a custom fork of Heretic, which is no longer public. So the numbers come from a standalone port of Heretic's evaluator, run in transformers with the weights dequantized, so the FP8 activation rounding in the Mamba projections is not part of the measured KL. Why thinking off, and what the refusal count means, is on the bf16 card.

Provenance: the original ThinkingCap refuses 99/100 with thinking off; the bf16 abliteration brings that to 21/100 at KL 0.0458 against the original; this quantization adds a further KL of 0.0148 relative to the abliterated model.

How it was made

Produced with llm-compressor 0.14.0 in the split NVIDIA uses for its own NVFP4 release of the base model: the experts as NVFP4A16 (FP4 (E2M1) weights in groups of 16, an FP8 (E4M3) scale per group, one FP32 global scale per tensor, bf16 activations), the Mamba2 projections as FP8_DYNAMIC (FP8 weights per channel, FP8 activations per token), attention in bf16.

The weights were quantized with GPTQ (llm-compressor's GPTQModifier, default settings). Calibration: the first 256 conversations of HuggingFaceH4/ultrachat_200k (train_sft), rendered in this model's chat template and cut to 2,048 tokens. A routed expert only sees the tokens the router sends it, so rarely chosen experts are quantized from fewer samples.

What is quantized

The routed experts and the shared expert of each of the 23 MoE layers (FP4), and the in_proj / out_proj of the 23 Mamba2 layers (FP8): 5,980 Linears. The q/k/v/o_proj of the 6 attention layers stay bf16.

Kept in bf16:

Fused projections

The experts are not gated (up_proj and down_proj only) and attention stays bf16, so no pair of layers needs a shared global scale.

Recipe

In plain llm-compressor 0.14 (transformers 5.17 stores the routed experts as fused 3D tensors; load_context splits them into per-expert Linears so the targets reach them, and the save restores the layout):

from transformers import AutoModelForCausalLM
from compressed_tensors.quantization.quant_scheme import preset_name_to_scheme
from llmcompressor import oneshot
from llmcompressor.modifiers.gptq import GPTQModifier
from llmcompressor.utils import load_context
with load_context(AutoModelForCausalLM):
    model = AutoModelForCausalLM.from_pretrained("<bf16-dir>", dtype="bfloat16")
experts = [r"re:.*mixer\.experts\.\d+\.(up|down)_proj$",
           r"re:.*mixer\.experts\.\d+\.gate_proj$",       # matches nothing; vLLM's MoE scheme lookup expects it
           r"re:.*mixer\.shared_experts\.(up|down)_proj$"]
recipe = GPTQModifier(
    config_groups={"experts_nvfp4a16": preset_name_to_scheme("NVFP4A16", experts),
                   "mamba_fp8_dynamic": preset_name_to_scheme("FP8_DYNAMIC", [r"re:.*mixer\.(in_proj|out_proj)$"])},
    ignore=["lm_head", "re:^mtp.*", r"re:.*mixer\.(q|k|v|o)_proj$"])
# ds: 256 ultrachat_200k train_sft conversations, rendered with the chat template, at most 2048 tokens each
oneshot(model=model, dataset=ds, recipe=recipe, max_seq_length=2048, num_calibration_samples=256)
model.save_pretrained("<out-dir>", save_compressed=True)
# llm-compressor 0.14 does not keep mtp.* for this model: copy the mtp.* tensors from the bf16
# checkpoint into the output and add "re:^mtp.*" to quantization_config.ignore

Not bit-reproducible: llm-compressor 0.14 draws the calibration samples in an unseeded random order, so re-running the same recipe gives slightly different files.

Limitations

Everything from the bf16 card applies: no safety filter, 21/100 harmful prompts still match Heretic's broad refusal markers with thinking off (1/100 with the abliteration's own, narrower list). You are the safety layer.

Add the usual 4-bit caveats: slightly lower accuracy on long multi-step reasoning and code than FP8. If a task works on FP8 and fails here, the cause is the quantization.

License

OpenMDW License Agreement 1.1, inherited from ThinkingCap and the NVIDIA base model (see LICENSE).

Credits

bottlecapai (ThinkingCap) · trohrbaugh (abliteration) · p-e-w/heretic · vllm-project/llm-compressor · NVIDIA (Nemotron 3.5 Lightning, and the quantization layout)

Calibration data: HuggingFaceH4/ultrachat_200k, MIT. No dataset text is included in this repo.