license: other
license_name: openmdw-1.1
license_link: LICENSE
base_model: IstroSec/ThinkingCap-Nemotron-3.5-Lightning-30B-A3B-abliterated
base_model_relation: quantized
library_name: transformers
pipeline_tag: text-generation
language:

• en
  tags:
• nemotron_h
• nemotron-3.5
• thinkingcap
• abliterated
• uncensored
• moe
• mamba
• fp8
• compressed-tensors
• llmcompressor
• vllm
• mtp

ThinkingCap-Nemotron-3.5-Lightning-30B-A3B-abliterated-FP8-DYNAMIC

An FP8 quantization (FP8 weights, dynamic per-token FP8 activations) of ThinkingCap-Nemotron-3.5-Lightning-30B-A3B-abliterated, the uncensored variant of bottlecapai/ThinkingCap-NVIDIA-Nemotron-3.5-Lightning-30B-A3B.

35.0 GB, down from 65.8 GB in bf16, at KL 0.0134 against the bf16 model. FP8 compute needs Ada, Hopper or Blackwell GPUs (sm89+); only a DGX Spark was tested.

Which build? The NVFP4A16 build (22.1 GB, KL 0.0148) and the W4A16 build (20.5 GB, KL 0.0215) are 4-bit weight-only and also run on Ampere. There is no FP8 build with 128×128 weight blocks: the experts' width (1856) is not a multiple of 128, and vLLM 0.30 refuses such MoE weights.

Usage

vLLM

Tested 2026-10-23 with vLLM 0.30.0 (vllm/vllm-openai:v0.30.0) on a DGX Spark (GB10), --max-model-len 32768 --max-num-seqs 8, MTP on (5 speculative tokens). Four requests at temperature 0 (a short explanation, a code task, a tool call, one request with thinking off) gave correct answers and a well-formed get_weather tool call.

vllm serve IstroSec/ThinkingCap-Nemotron-3.5-Lightning-30B-A3B-abliterated-FP8-DYNAMIC --async-scheduling \
  --reasoning-parser nemotron_v3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}'

Sampling (NVIDIA's recommendation for the base model, which ThinkingCap uses unchanged): temperature 1.0, top_p 0.95. Thinking is on by default; turn it off per request with chat_template_kwargs: {"enable_thinking": false}.

MTP. The head is bf16 and listed in quantization_config.ignore. In the test above (5 speculative tokens, temperature 0, 4 short requests) vLLM accepted 58% of drafted tokens (870 of 1,495, mean acceptance length 3.91); figures from sampling-based runs are not directly comparable.

Memory. On GPUs with less memory, vLLM can stop at startup with max_num_seqs (1024) exceeds available Mamba cache blocks; lower --max-num-seqs.

Transformers

Loads through llm-compressor's loader, with the weights dequantized to bf16 (about 66 GB of memory):

from transformers import AutoModelForCausalLM, CompressedTensorsConfig
from llmcompressor.utils import load_context
with load_context(AutoModelForCausalLM):
    m = AutoModelForCausalLM.from_pretrained("IstroSec/ThinkingCap-Nemotron-3.5-Lightning-30B-A3B-abliterated-FP8-DYNAMIC", device_map="cuda",
                                             quantization_config=CompressedTensorsConfig(dequantize=True))

llama.cpp

Not for llama.cpp: this is compressed-tensors format. Use the GGUF builds.

Evaluation

Heretic's evaluation protocol, 100 harmful and 100 harmless prompts, against the bf16 abliterated checkpoint, with thinking off:

Decoding is greedy, but the batched GPU kernels are not bit-deterministic here: two runs of the bf16 weights gave 21 and 23 refusals. Differences of a few refusals between builds are not significant.

Heretic cannot load this architecture; the numbers come from a standalone port of its evaluator, run in transformers with the weights dequantized, so FP8 activation rounding is not part of the measured KL. Why thinking off, and what the refusal count means, is on the bf16 card.

Provenance: the original ThinkingCap refuses 99/100 with thinking off; the bf16 abliteration brings that to 21/100 at KL 0.0458 against the original; this quantization adds a further KL of 0.0134 relative to the abliterated model.

How it was made

Produced with llm-compressor 0.14.0, scheme FP8_DYNAMIC: FP8 (E4M3) weights with one scale per output channel, and FP8 activations with a scale computed per token at run time. Data-free: no calibration data.

What is quantized

Every Linear in the decoder: the 128 routed experts and the shared expert of each of the 23 MoE layers, the in_proj / out_proj of the 23 Mamba2 layers, and q/k/v/o_proj of the 6 attention layers (6,004 Linears).

Kept in bf16:

Recipe

In plain llm-compressor 0.14 (transformers 5.17 stores the routed experts as fused 3D tensors; load_context splits them into per-expert Linears so targets="Linear" reaches them, and the save restores the layout):

from transformers import AutoModelForCausalLM
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.utils import load_context
with load_context(AutoModelForCausalLM):
    model = AutoModelForCausalLM.from_pretrained("<bf16-dir>", dtype="bfloat16")
recipe = QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head", "re:^mtp.*"])
oneshot(model=model, recipe=recipe)
model.save_pretrained("<out-dir>", save_compressed=True)
# llm-compressor 0.14 does not keep mtp.* for this model: copy the mtp.* tensors from the bf16
# checkpoint into the output and add "re:^mtp.*" to quantization_config.ignore

Limitations

Everything from the bf16 card applies: no safety filter, 21/100 harmful prompts still match a refusal marker with thinking off. You are the safety layer.

License

OpenMDW License Agreement 1.1, inherited from ThinkingCap and the NVIDIA base model (see LICENSE).

Credits

bottlecapai (ThinkingCap) · trohrbaugh (abliteration) · p-e-w/heretic · vllm-project/llm-compressor · NVIDIA (Nemotron 3.5 Lightning)