license: other
license_name: polyform-small-business-1.0.0
license_link: LICENSE
base_model: IstroSec/ThinkingCap-Qwen3.8-27B-abliterated
base_model_relation: quantized
library_name: transformers
pipeline_tag: image-text-to-text
tags:

• qwen3_5
• autoround
• w4a16
• int4
• compressed-tensors
• vllm
• marlin
• mtp
• vision
• abliterated
• uncensored

ThinkingCap-Qwen3.8-27B-abliterated — W4A16 AutoRound

4-bit weight-only quantization (W4A16, int4 symmetric, group_size 128) of
IstroSec/ThinkingCap-Qwen3.8-27B-abliterated
using AutoRound via llm-compressor 0.14 + auto-round 0.15.

• Quantized: all language-model Linear layers (400 modules: attention, MLP and GatedDeltaNet projections) → compressed-tensors pack-quantized (served by vLLM's Marlin kernels).
• Preserved in BF16 (untouched, bit-identical to the base model):
  • Vision tower (model.visual.*, 333 tensors)
  • MTP / Multi-Token Prediction head (mtp.*, 15 tensors) — enables speculative decoding
  • lm_head, embeddings, norms, conv1d, and the GatedDeltaNet in_proj_a/in_proj_b projections (output dim 48 < group_size 128)
• Calibration: 128 samples × 2048 tokens from neuralmagic/LLM_compression_calibration with the model's own chat template, 200 AutoRound iterations per layer.
• Size: ~19.5 GB (base BF16: ~52 GB).

Quality

Perplexity measured on 50 held-out texts from wikitext-103, 16,007 evaluated tokens.
Deliberately not the calibration set, which would flatter the result. Same texts, same
tokenization (identical token count for both), same method (raw text, no chat template),
both models served by vLLM:

This is above the ~5% degradation usually accepted for W4A16 (int4, group_size 128),
so treat the LM fidelity as degraded more than a typical W4A16 quant of this size. The
degradation is not visible in short side-by-side chat samples, which read as equivalent to
the BF16 original — perplexity averages small probability shifts over every token and is
the more sensitive instrument here. If you need tighter fidelity, re-quantizing with more
AutoRound iterations (iters=400) and more calibration samples (nsamples=256) is the
first knob to turn.

Note that perplexity measures language-modelling fidelity only; it is not a substitute for
task-specific evaluation.

Usage (vLLM ≥ 0.28)

vllm serve DoktorMincs/ThinkingCap-Qwen3.8-27B-abliterated-W4A16-AutoRound \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --reasoning-parser qwen3 \
  --enable-prefix-caching \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'

• Vision inputs work normally (image/video); the vision tower runs in BF16.
• The MTP draft head loads from the same checkpoint for speculative decoding.
• Validated on vLLM 0.28 (text, MTP speculative decoding, vision), Marlin kernels
  (CompressedTensorsWNA16).
• This is a thinking model: with a small max_tokens the whole budget can be consumed
  by the reasoning trace, leaving the visible answer empty. Use max_tokens ≥ 1024 and
  prefer the chat endpoint over raw completion.

Notes

• Architecture: Qwen3_5ForConditionalGeneration (hybrid: 64 layers, 3:1 GatedDeltaNet
  linear-attention : full-attention, 27-block ViT, 1 MTP layer).
• Abliteration is inherited from the base model — quantizing does not remove it, and this
  derivative carries the same behavioural characteristics and the same usage warnings.
• Calibration data is general (math/code/logic/science QA); for heavy RP/creative use, a
  domain-matched calibration set may further improve fidelity.

License

Inherited from the base model: PolyForm Small Business License 1.0.0, © 2026 BottleCap AI.
See LICENSE and NOTICE in this repository.