license: apache-2.0
library_name: transformers
pipeline_tag: image-text-to-text
base_model: DavidAU/Qwen3.8-27B-Ultimate-Heretic-Uncensored
tags:
- qwen
- qwen3_8
- finetune
- awq
- int4
- w4a16
- compressed-tensors
- vllm
- coding-agent
- agentic
Qwen3.8-27B-Ultimate-Heretic-Uncensored-W4A16-AWQ
W4A16-AWQ quantization of DavidAU/Qwen3.8-27B-Ultimate-Heretic-Uncensored (55.7 GB -> 19.6 GB).
See the GGUF card for all prompting instructions.
Quantization method
| Layer | Format | Group |
|---|---|---|
| MLP, attention (q/k/v/o, in_proj_qkv/z, out_proj) | W4A16_ASYM (int4) | 128 |
| in_proj_a/b | BF16 | — |
| lm_head, embed_tokens | BF16 | — |
| visual, mtp, norms | BF16 | — |
- Tooling: llmcompressor
oneshotwith CPU offloading +AWQModifier(hybrid-attention mappings, linear_attn rules dropped). - Run command:
CUDA_VISIBLE_DEVICES=1,2 python3 quantize-awq-hybrid.py \
--model_path ./Qwen3.8-27B-Ultimate-Heretic-Uncensored \
--quant_path ./Qwen3.8-27B-Ultimate-Heretic-Uncensored-W4A16-AWQ
Recommended parameters
From the base model: temperature 1.0, top_p 0.95, top_k 20, min_p 0, presence_penalty 0, repetition_penalty 1.0.
Usage
Tested with 1Cat-vLLM (compressed-tensors, SM70 kernels):
from vllm import LLM, SamplingParams
llm = LLM(
"TheUnderscore/Qwen3.8-27B-Ultimate-Heretic-Uncensored-W4A16-AWQ",
tensor_parallel_size=2,
max_model_len=8192,
)
output = llm.generate("Hello, who are you?", SamplingParams(temperature=1.0, top_p=0.95, max_tokens=256))
print(output[0].text)
License
Apache-2.0, the same as the finetune.
Files
quantize-awq-hybrid.py— the script used to produce this quantization (CPU-offloaded, DDP/torchrun, produces properly numbered-of-Nshards).