library_name: ninfer
pipeline_tag: image-text-to-text
license: other
license_name: polyform-small-business-1.0.0-with-bottlecap-personal-use-permission
license_link: https://huggingface.co/Dragoy/ThinkingCap-Qwen3.8-27B-abliterated-NVFP4-NInfer/blob/main/LICENSE
base_model: bottlecapai/ThinkingCap-Qwen3.8-27B
tags:
- ninfer
- nvfp4
- fp8
- qwen3.8
- qwen3_5
- abliterated
- multimodal
- mtp
- dflash2
- speculative-decoding
- blackwell
- sm_120a
ThinkingCap-Qwen3.8-27B · Huihui-style weight projection · NVFP4 · NInfer v3
A downloadable single-file, 27.78B-parameter multimodal model derived from BottleCap AI's ThinkingCap-Qwen3.8-27B, with a refusal-direction weight-space approximation, mixed NVFP4/FP8 quantization, and the official NInfer v3 container. Includes text, vision, MTP and DFlash2. The gated original checkpoint is not included in BF16 form; the .ninfer file contains the resulting runnable weights. This is huihui-style, not a checkpoint from the huihui-ai organization and not a transfer of their published weight differences.
| Verified release | |
|---|---|
| Base | bottlecapai/ThinkingCap-Qwen3.8-27B @ 52361b720e30f8835af8e401d876ec3077b4a823 |
| Transformation | One normalized refusal direction; BF16 weight-space rank-one approximation on 131 tensors. Not equivalent to hooks after every decoder layer. |
| Quantization | NVFP4 for MLP layers 0–55, FP8 for attention/GDN and MLP layers 56–63; other tensors retain the precisions listed in the inspection report. 32 UltraChat calibration examples, 2048-token maximum. |
| Converter/runtime | Neroued/ninfer @ bace20dc70249eed6402b66d4852c6c3f9612905, official qwen3_8_27b_nvfp4 recipe, CUDA 13.0, compiled for Blackwell sm_120a |
| Speculative drafter | z-lab/Qwen3.8-27B-DFlash2 @ 50307d4c4cde6860d4eee73e2547cd786fe8e8a4, imported as a separate DFlash2 component |
| Download | qwen3_8_27b_thinkingcap_huihui_nvfp4.ninfer, 23,719,719,172 bytes, SHA-256 e2be7fa2f0a7bf99b919ec1ba1fe2f2808ad29a217ecca27c3fa94c0be5a1de2 |
| Inspector | Container v3; 1,246 objects: 1,240 tensors, 6 resources; 1,513 bindings and 844 uses. Components text,vision,mtp,dflash2. |
Run it
Requires an NVIDIA Blackwell GPU with an sm_120a NInfer build. Tested on a Modal NVIDIA RTX PRO 6000 Blackwell Server Edition. Build the pinned engine and download the artifact:
git clone https://github.com/Neroued/ninfer.git
cd ninfer
git checkout bace20dc70249eed6402b66d4852c6c3f9612905
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=120a
cmake --build build -j8
cd ..
hf download Dragoy/ThinkingCap-Qwen3.8-27B-abliterated-NVFP4-NInfer \
qwen3_8_27b_thinkingcap_huihui_nvfp4.ninfer --local-dir model
MODEL="$PWD/model/qwen3_8_27b_thinkingcap_huihui_nvfp4.ninfer"
ninfer/build/apps/ninfer "$MODEL" --prompt "Reply with one short greeting." \
--max-context 8192 --max-new 48 --kv-dtype fp8 --no-thinking --greedy
ninfer/build/apps/ninfer "$MODEL" --prompt "Name a planet in one sentence." \
--max-context 8192 --max-new 48 --kv-dtype fp8 --no-thinking --greedy \
--spec mtp --draft-tokens 3 --lm-head-draft
ninfer/build/apps/ninfer "$MODEL" --prompt "Name a moon in one sentence." \
--max-context 8192 --max-new 48 --kv-dtype fp8 --no-thinking --greedy \
--spec dflash2 --draft-tokens 7 --lm-head-draft
(cd ninfer && ./build/apps/ninfer "$MODEL" \
--messages examples/cli/messages/image_chart.json --vision \
--max-context 8192 --max-new 48 --kv-dtype fp8 --no-thinking --greedy)
The vision example references the chart image inside the pinned NInfer checkout. The .ninfer container embeds the original ThinkingCap tokenizer, chat template, generation configuration, image and video preprocessors. DFlash2 was trained for a different Qwen target; the target verifies its proposals. Draft acceptance, speedup, long-context behavior and output quality have not been established for this derivative.
What was changed
The direction is a unit FP32 vector of width 5,120, computed from the difference of mean last-token residuals of 32 refusal and 32 benign calibration instructions at decoder layer 37 (60% depth), using the procedure inspired by Sumandora/remove-refusals-with-transformers @ 7786b0a. The frozen BF16 source, sampling partitions and direction checks are recorded in huihui-method-report.json; the vector is recipe/huihui-direction.safetensors. No calibration prompts or harmful generations are in this repository.
For unit direction $d$, embedding rows are projected as $e' = e - (e\cdot d)d$ and residual-producing output matrices as $W' = W - d(d^\top W)$ (the vision merger output bias is projected too), rounded back to BF16. The transformed set is: token embeddings; attention/GDN outputs and MLP down projections in each of the 64 language layers; vision merger linear_fc2.weight and .bias. Other vision tower weights and MTP weights are not abliterated. The transform is not mathematically identical to the earlier temporary inference hooks after each decoder layer: residual paths and nonlinearities intervene. Results measured on the hook-based BF16 experiment must not be attributed to these materialized or quantized weights.
The original 18-shard BF16 checkpoint was copied and verified, not overwritten. See huihui-materialization.json for each changed/unchanged shard and tensor fingerprints and recipe/huihui_materialize.py for the implementation. NVFP4/FP8 quantization used the pinned reference script and allocation from Dragoy/Swift-Qwen3.8-27B-abliterated-NVFP4-NInfer @ 4c25aa2, with torch 2.11.0, transformers 5.10.1, llmcompressor 0.12.0.1 and compressed-tensors 0.17.1. Observed UltraChat revision: 8049631c405ae6576f93f445c6b8166f76f5505a; the reference calibration script does not pin the dataset revision. Quantized output contains text, while vision/MTP come from the modified BF16 checkpoint. To resolve the quantizer's original model.language_model.* tensor names, a metadata-only NInfer input adapter adds the source text_config to a linked copy of its config; encoded weight bytes are unchanged. See recipe/ninfer_convert.py, huihui-conversion-report.json and the conversion record for exact SHAs, mappings and source revisions.
The official converter was run with --components text,vision,mtp,dflash2 --proposal. Its writer requires POSIX hard links, unavailable on Modal Volume, so conversion and v3 inspection ran on the container's temporary POSIX disk. The artifact was then copied byte-for-byte to the persistent volume with a second SHA-256 verification. Only the temporary output path in the published conversion report was normalized to the persistent path; the original upstream report SHA is retained in huihui-conversion-report.json. Check all published file digests in SHA256SUMS.
Measured on this artifact
| Smoke check | Observed result |
|---|---|
| NInfer inspector | v3, exact four-component inventory above and expected NVFP4/FP8 allocations |
| Text, greedy, 48-token cap | Hello! |
| Vision using NInfer example chart | NIFER VISION 731；3；左侧 |
| MTP speculative path | Mars is the fourth planet from the Sun. |
| DFlash2 speculative path | Europa is a moon of Jupiter known for its icy surface and suspected subsurface ocean. |
These are one benign prompt per path, not representative evaluations. The runtime report contains the invocations' outputs and logs. No refusal rate, HarmBench ASR, paired helpfulness, coding benchmark, capability retention, draft acceptance rate or quality tolerance has been measured on these published weights. In particular, stopped or incomplete HarmBench runs on the original BF16 model are not model scores for this artifact. Abliteration can change both refusal and useful behavior. Users deploying this less-restrictive derivative must implement their own safeguards and comply with applicable law.
License and rights
The source ThinkingCap weights are not Apache-only. This derivative retains BottleCap AI's original LICENSE, NOTICE, PolyForm Small Business License 1.0.0 and the Qwen upstream Apache-2.0 license. Required Notice: Copyright 2026 BottleCap AI (https://bottlecapai.com) remains applicable. DFlash2 is separately distributed under its upstream Apache-2.0 terms; credit z-lab's original model. No sublicense of BottleCap AI rights is offered by this model repository.
BottleCap AI adds a license for individual personal, noncommercial use of ThinkingCap; PolyForm's permitted small-business purposes have their own size/revenue conditions. The additional individual permission expressly says “use,” not “publicly redistribute derivatives”; public redistribution by an individual is not expressly addressed there. This repository is published by an individual under their personal, noncommercial interpretation, not as a representation that anyone may redistribute the weights or use them commercially. Consult the original terms and BottleCap AI about any uncertain distribution or commercial rights. This description is not legal advice. The artifact and outputs are provided as-is; no safety or fitness warranty.