license: other
license_name: polyform-small-business-1.0.0
license_link: LICENSE
base_model: IstroSec/ThinkingCap-Qwen3.8-27B-abliterated
base_model_relation: quantized
library_name: transformers
pipeline_tag: image-text-to-text
language:
- en
- multilingual
tags: - qwen3_5
- qwen3_8
- thinkingcap
- abliterated
- uncensored
- nvfp4
- nvfp4a16
- fp4
- compressed-tensors
- llmcompressor
- vllm
- mtp
- vision
ThinkingCap-Qwen3.8-27B-abliterated-NVFP4A16
NVFP4 weight-only quantization (4-bit FP4 weights, bf16 activations) of ThinkingCap-Qwen3.8-27B-abliterated, the uncensored variant of bottlecapai/ThinkingCap-Qwen3.8-27B.
28.6 GB (from 55.6 GB bf16). Plan on a 48 GB-class GPU; 32 GB leaves almost nothing for KV cache. Fastest on Blackwell (native FP4 path); runs on Hopper via the Marlin weight-only kernel (the same support matrix bottlecapai lists for its own NVFP4 weight-only build).
If you have the memory for it, the FP8-DYNAMIC build is closer to lossless. This one is for when 36.8 GB doesn't fit.
What is quantized
Produced with llm-compressor, scheme NVFP4A16: FP4 (E2M1) weights in groups of 16 with FP8 (E4M3) per-group scales and a per-tensor FP32 global scale. Activations stay bf16, so no calibration set is needed and there's no activation-quantization noise.
Quantized: all Linear modules in the 64 decoder layers' MLPs and the 16 full-attention layers' q/k/v/o_proj.
Kept in bf16 on purpose:
| Component | Why |
|---|---|
linear_attn.* (Gated DeltaNet, 48 layers) |
4-bit on the recurrent block roughly doubles KL and introduces thinking loops on this architecture |
visual.* |
vision tower and merger |
lm_head |
standard |
mtp.* |
MTP head, re-grafted from bf16 after quantization; its Linears are in quantization_config.ignore |
The bf16 islands are why this lands at 28.6 GB rather than the ~21 GB of a build that also quantizes DeltaNet (bottlecapai's own NVFP4 weight-only is 21 GB). The DeltaNet block is 5.6B of the 28B parameters. Community measurements on the Qwen3.6-27B sibling showed that 4-bit-quantizing it roughly doubles KL and can produce thinking loops, so those 8 GB are deliberately spent. If size matters more than that risk, bottlecapai's build of the original (censored) model shows the smaller trade-off.
Evaluation
| Refusals (100 harmful) | KL vs. bf16 abliterated | |
|---|---|---|
| bf16 abliterated (source) | 6 / 100 | — |
| FP8-DYNAMIC | {{fp8.refusals}} | {{fp8.kl}} |
| NVFP4A16 (this repo) | {{nvfp4a16.refusals}} | {{nvfp4a16.kl}} |
Method: Heretic --evaluate-model against the bf16 abliterated checkpoint, non-thinking mode. Expect this build to land somewhat above FP8; under ~0.05 KL is the target for a 4-bit build of a reasoning model. {{nvfp4a16.kl_note}}
Provenance chain: original ThinkingCap refuses 97/100 → bf16 abliteration 6/100 at KL 0.065 vs. original → this quantization adds the KL above.
Usage
vLLM (Blackwell for the native FP4 path; Hopper falls back to Marlin automatically)
vllm serve IstroSec/ThinkingCap-Qwen3.8-27B-abliterated-NVFP4A16 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice --tool-call-parser qwen3_xml \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
--max-model-len 65536 --gpu-memory-utilization 0.85
Sampling (Qwen3.8 recommendations, which ThinkingCap uses unchanged): thinking mode temperature 1.0, top_p 0.95, top_k 20, min_p 0; non-thinking mode temperature 0.7, top_p 0.8, top_k 20, presence_penalty 1.5. Thinking budget via chat_template_kwargs: {"reasoning_effort": "xhigh"} — xhigh (default, recommended), medium, or low.
Notes:
- Only the 16 full-attention layers have a KV cache; FP8 KV (
--kv-cache-dtype fp8_e5m2) is a small win here, not a large one. - Weight-only FP4 is bandwidth-bound like any weight-only format: single-stream decode is fast, high-batch throughput is where W4A4 (
NVFP4) would pull ahead. That variant needs calibration and loses more quality; it isn't published here.
Transformers loads it (dequantized to bf16, so ~56 GB of memory):
from transformers import AutoModelForImageTextToText
m = AutoModelForImageTextToText.from_pretrained("IstroSec/ThinkingCap-Qwen3.8-27B-abliterated-NVFP4A16", device_map="cuda")
Not for llama.cpp — compressed-tensors format. GGUF builds are made separately from the bf16 source.
Reproduce
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
recipe = QuantizationModifier(
targets="Linear", scheme="NVFP4A16",
ignore=["lm_head", "re:.*visual.*", "re:.*linear_attn.*"],
)
oneshot(model=model, recipe=recipe)
# then copy mtp.* from the bf16 checkpoint and add the MTP Linears to quantization_config.ignore
Limitations
Everything from the bf16 card applies: no safety filter, 6/100 residual refusals, thinking mode not separately evaluated. You are the safety layer. Add to that the usual 4-bit caveats: slightly lower accuracy on long multi-step reasoning and code than FP8; if a task is failing here and working on FP8, that's the quantization.
License
PolyForm Small Business License 1.0.0 + BottleCap personal-use grant, inherited from ThinkingCap (see LICENSE). Upstream Qwen materials and the abliteration adapter are Apache-2.0 (see NOTICE). Commercial use beyond the PolyForm terms: contact BottleCap AI.
Credits
bottlecapai (ThinkingCap) · MuXodious (abliteration adapter) · p-e-w/heretic · vllm-project/llm-compressor · Qwen team