license: apache-2.0
library_name: transformers
pipeline_tag: image-text-to-text
base_model: DavidAU/Qwen3.8-27B-Ultimate-Heretic-Uncensored
tags:
- qwen
- qwen3_8
- finetune
- nvfp4
- w4a16
- awq
- mixed-precision
- compressed-tensors
- vllm
- coding-agent
- agentic
Qwen3.8-27B-Ultimate-Heretic-Uncensored-NVFP4-W4ATTN-AWQ
Mixed-precision quantization of DavidAU/Qwen3.8-27B-Ultimate-Heretic-Uncensored (55.6 GB -> 20.3 GB).
See the GGUF card for all prompting instructions.
Quantization method
| Layer | Format | Group | Why |
|---|---|---|---|
| MLP (gate/up/down) | NVFP4 (float4) | 16 | High within-group variance needs fine scale resolution |
| Attention (q/k/v/o, in_proj_qkv/z, out_proj) | W4A16_ASYM (int4) | 128 | Lower within-group variance; AWQ smoothing rebalances channel scales |
| in_proj_a/b | BF16 | — | 48-wide delta-rule gating; int4 breaks recurrence |
| lm_head, embed_tokens | BF16 | — | Output embeddings dominate perplexity |
| visual, mtp, norms | BF16 | — | vision not group-divisible; mtp/norms too small |
- Tooling: llmcompressor
oneshotwithconfig_groups(per-layer dispatch) +AWQModifier(hybrid-attention mappings, linear_attn rules dropped). - Output:
quant_method: "compressed-tensors", two config_groups (W4A16int4 +NVFP4float4). Loadable by 1Cat-vLLM (SM70 Marlin/TurboMind kernels for both schemes). - Run command:
CUDA_VISIBLE_DEVICES=1,2 python3 quantize-nvfp4-hybrid-fp16.py \
--model_path ./Qwen3.8-27B-Ultimate-Heretic-Uncensored \
--quant_path ./Qwen3.8-27B-Ultimate-Heretic-Uncensored-NVFP4-W4ATTN-AWQ \
--awq
Recommended parameters
From the base model: temperature 1.0, top_p 0.95, top_k 20, min_p 0, presence_penalty 0, repetition_penalty 1.0.
Usage
Tested with 1Cat-vLLM (compressed-tensors config_groups dispatch, SM70 kernels):
from vllm import LLM, SamplingParams
llm = LLM(
"TheUnderscore/Qwen3.8-27B-Ultimate-Heretic-Uncensored-NVFP4-W4ATTN-AWQ",
tensor_parallel_size=2,
max_model_len=8192,
)
output = llm.generate("Hello, who are you?", SamplingParams(temperature=1.0, top_p=0.95, max_tokens=256))
print(output[0].text)
License
Apache-2.0, the same as the finetune.
Files
quantize-nvfp4-hybrid-fp16.py— the script used to produce this quantization (CPU-offloaded, DDP/torchrun,config_groupsmixed-precision, pre-flight coverage check).model-nonquant.safetensors— unquantized tensors (mtp.*andmodel.visual.*) preserved BF16.