license: other
license_name: openmdw-1.1
license_link: LICENSE
base_model: IstroSec/ThinkingCap-Nemotron-3.5-Lightning-30B-A3B-abliterated
base_model_relation: quantized
library_name: transformers
pipeline_tag: text-generation
language:

• en
  tags:
• nemotron_h
• nemotron-3.5
• thinkingcap
• abliterated
• uncensored
• moe
• mamba
• int4
• w4a16
• gptq
• compressed-tensors
• llmcompressor
• vllm
• mtp
  datasets:
• HuggingFaceH4/ultrachat_200k

ThinkingCap-Nemotron-3.5-Lightning-30B-A3B-abliterated-W4A16

A 4-bit integer weight-only quantization (INT4 weights in groups of 64, bf16 activations) of ThinkingCap-Nemotron-3.5-Lightning-30B-A3B-abliterated, the uncensored variant of bottlecapai/ThinkingCap-NVIDIA-Nemotron-3.5-Lightning-30B-A3B.

20.5 GB, down from 65.8 GB in bf16, at KL 0.0215 against the bf16 model. For GPUs without FP4 or FP8 support, such as Ampere. vLLM serves it on Ampere and newer (sm80+); only a DGX Spark was tested.

Which build? The NVFP4A16 build (22.1 GB, KL 0.0148) runs on the same GPUs in vLLM and diverges less for 1.6 GB more; with an Ada or newer GPU and the memory, FP8-DYNAMIC (35.0 GB, KL 0.0134). Speed was not measured.

Usage

vLLM

Tested 2026-10-23 with vLLM 0.30.0 (vllm/vllm-openai:v0.30.0) on a DGX Spark (GB10), --max-model-len 32768 --max-num-seqs 8, MTP on (5 speculative tokens). Four requests at temperature 0 (a short explanation, a code task, a tool call, one request with thinking off) gave correct answers and a well-formed get_weather tool call.

vllm serve IstroSec/ThinkingCap-Nemotron-3.5-Lightning-30B-A3B-abliterated-W4A16 --async-scheduling \
  --reasoning-parser nemotron_v3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}'

Sampling (NVIDIA's recommendation for the base model, which ThinkingCap uses unchanged): temperature 1.0, top_p 0.95. Thinking is on by default; turn it off per request with chat_template_kwargs: {"enable_thinking": false}.

MTP. The head is bf16 and listed in quantization_config.ignore. In the test above (5 speculative tokens, temperature 0, 4 short requests) vLLM accepted 59% of drafted tokens (820 of 1,400, mean acceptance length 3.93); figures from sampling-based runs are not directly comparable.

Memory. On GPUs with less memory, vLLM can stop at startup with max_num_seqs (1024) exceeds available Mamba cache blocks; lower --max-num-seqs.

Transformers

Loads through llm-compressor's loader, with the weights dequantized to bf16 (about 66 GB of memory):

from transformers import AutoModelForCausalLM, CompressedTensorsConfig
from llmcompressor.utils import load_context
with load_context(AutoModelForCausalLM):
    m = AutoModelForCausalLM.from_pretrained("IstroSec/ThinkingCap-Nemotron-3.5-Lightning-30B-A3B-abliterated-W4A16", device_map="cuda",
                                             quantization_config=CompressedTensorsConfig(dequantize=True))

llama.cpp

Not for llama.cpp: this is compressed-tensors format. Use the GGUF builds.

Evaluation

Heretic's evaluation protocol, 100 harmful and 100 harmless prompts, against the bf16 abliterated checkpoint, with thinking off:

Decoding is greedy, but the batched GPU kernels are not bit-deterministic here: two runs of the bf16 weights gave 21 and 23 refusals. Differences of a few refusals between builds are not significant.

Heretic cannot load this architecture; the numbers come from a standalone port of its evaluator, run in transformers with the weights dequantized. Why thinking off, and what the refusal count means, is on the bf16 card.

Provenance: the original ThinkingCap refuses 99/100 with thinking off; the bf16 abliteration brings that to 21/100 at KL 0.0458 against the original; this quantization adds a further KL of 0.0215 relative to the abliterated model.

How it was made

Produced with llm-compressor 0.14.0, scheme W4A16 with the group size changed to 64: symmetric INT4 weights with one bf16 scale per 64 weights. The preset's group of 128 does not divide the experts' width (1856 = 14.5 × 128); 64 does. Activations stay bf16.

The weights were quantized with GPTQ (llm-compressor's GPTQModifier, default settings). GPTQ needs activations, so the build is calibrated: the first 256 conversations of HuggingFaceH4/ultrachat_200k (train_sft), rendered in this model's chat template and cut to 2,048 tokens. A routed expert only sees the tokens the router sends it, so rarely chosen experts are quantized from fewer samples.

What is quantized

Every Linear in the decoder: the 128 routed experts and the shared expert of each of the 23 MoE layers, the in_proj / out_proj of the 23 Mamba2 layers, and q/k/v/o_proj of the 6 attention layers (6,004 Linears).

Kept in bf16:

Recipe

In plain llm-compressor 0.14 (transformers 5.17 stores the routed experts as fused 3D tensors; load_context splits them into per-expert Linears so targets="Linear" reaches them, and the save restores the layout):

from transformers import AutoModelForCausalLM
from compressed_tensors.quantization import QuantizationScheme, QuantizationArgs
from llmcompressor import oneshot
from llmcompressor.modifiers.gptq import GPTQModifier
from llmcompressor.utils import load_context
with load_context(AutoModelForCausalLM):
    model = AutoModelForCausalLM.from_pretrained("<bf16-dir>", dtype="bfloat16")
w4g64 = QuantizationScheme(targets=["Linear"], weights=QuantizationArgs(
    num_bits=4, type="int", symmetric=True, strategy="group", group_size=64))
recipe = GPTQModifier(config_groups={"group_0": w4g64}, ignore=["lm_head", "re:^mtp.*"])
# ds: 256 ultrachat_200k train_sft conversations, rendered with the chat template, at most 2048 tokens each
oneshot(model=model, dataset=ds, recipe=recipe, max_seq_length=2048, num_calibration_samples=256)
model.save_pretrained("<out-dir>", save_compressed=True)
# llm-compressor 0.14 does not keep mtp.* for this model: copy the mtp.* tensors from the bf16
# checkpoint into the output and add "re:^mtp.*" to quantization_config.ignore

Not bit-reproducible: llm-compressor 0.14 draws the calibration samples in an unseeded random order, so re-running the same recipe gives slightly different files.

Limitations

Everything from the bf16 card applies: no safety filter, 21/100 harmful prompts still match a refusal marker with thinking off. You are the safety layer.

Add the usual 4-bit caveats: slightly lower accuracy on long multi-step reasoning and code than FP8. If a task works on FP8 and fails here, the cause is the quantization.

License

OpenMDW License Agreement 1.1, inherited from ThinkingCap and the NVIDIA base model (see LICENSE).

Credits

bottlecapai (ThinkingCap) · trohrbaugh (abliteration) · p-e-w/heretic · vllm-project/llm-compressor · NVIDIA (Nemotron 3.5 Lightning)

Calibration data: HuggingFaceH4/ultrachat_200k, MIT. No dataset text is included in this repo.