license: apache-2.0
base_model:
- Altworld/Hemmingway-1
- Qwen/Qwen3.8-27B
pipeline_tag: text-generation
library_name: transformers
tags: - quantization
- fp8
- int8
- compressed-tensors
- w8a8
- abliteration
- uncensored
- qwen
Hemmingway-1-27B-abliterated — INT8 W8A8 (dynamic)
8-bit INT8-W8A8 of the 27.32B abliterated rewrite model — 54% of the BF16 original, language-model linears only.
Status: structurally verified and runnable, not benchmarked. See
Verification for exactly what was measured and what was not.
What this is
An 8-bit compressed-tensors quantization ofdistributedcognition/Hemmingway-1-27B-abliterated, produced withllmcompressor.model_free_ptq directly from the BF16 safetensors. The result is an ordinary
Hugging Face model directory whose linear weights are stored in 8-bit with compressed-tensors
metadata in config.json, so it loads in vLLM, SGLang, or Transformers without a conversion step.
| Architecture | Qwen3_5ForCausalLM |
| Parameters | 27.32B |
| Layers / hidden / vocab | 64 / 5120 / 248,320 |
| Source BF16 | 54.64 GB, 866 tensors |
| This artifact | 30.30 GB (55.5% of BF16) |
| Linear layers quantized | 496 |
| Weight format | INT8, per-output-channel scale, symmetric |
| Activation format | INT8, dynamic, per-token |
| Chat template | chat_template.jinja included |
Quantization scope
Only linear layers inside the language-model decoder were quantized. Everything else stays BF16:
lm_headandmodel.embed_tokens.- The MTP (multi-token-prediction) head — all 15
mtp.*tensors including its 8 linear weights, so speculative decoding still runs, at BF16. - The Gated DeltaNet state tensors:
conv1d(4D depthwise),A_log,dt_bias.
Why the state tensors are excluded: conv1d is a depthwise kernel that 8-bit channel-wise quantization mangles badly — measured at 1.0 max relative error against its own magnitude. A_log and dt_bias are 48-element decay vectors where absolute precision is the whole value.
Of 866 source tensors, 496 linear weights were quantized: attention projections across 48 Gated DeltaNet and 16 full-attention layers, plus MLP gate/up/down in all 64 layers.
Keeping the embedding at BF16 is not cosmetic. tie_word_embeddings=False, so lm_head and the embedding are separate tensors and both stay at full precision. The failure mode, measured directly,
is silent: the weights load without error, the file is a plausible size, and the model then returns
the same handful of low-ID tokens no matter what you feed it.
Each quantized layer stores a per-output-channel scale in weight_scale.
Usage
vLLM
vllm serve moeshawky/Hemmingway-1-27B-abliterated-INT8-W8A8
SGLang
python -m sglang.launch_server --model-path moeshawky/Hemmingway-1-27B-abliterated-INT8-W8A8
Transformers
from transformers import AutoModelForCausalLM, AutoTokenizer
name = "moeshawky/Hemmingway-1-27B-abliterated-INT8-W8A8"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, dtype="bfloat16", device_map="auto")
Hybrid attention. 48 of 64 layers use Gated DeltaNet linear attention and 16 use full attention. the recurrent state stays BF16, so long-context behaviour should match the source; the quantized part is the projections around it.
Verification
Measured:
- Weight fidelity. FP8 ≈ 31.6 dB and INT8 ≈ 39.6–40.8 dB SNR against the BF16 source across sampled self-attention, linear-attention, and MLP projections.
- Output distribution and generation. Last-position logits σ = 2.0469 (FP8) and 2.0781 (INT8) against 2.0625 for the BF16 source; near-identical top-5 and a word-for-word identical opening sentence (
the wind carried the smell of salt and rust) before natural argmax divergence. - Tensor census. 496 quantized tensors and 496 scales across 13 shards, with
lm_head,embed_tokens, all 15mtp.*tensors, and all SSM state tensors confirmed BF16. - Schemas.
config.jsondeclaresquant_method=compressed-tensors,format=int-quantized,
weights 8-bit /num_bits: 8/ channel strategy / symmetric, input activations 8-bit dynamic
per-token.
Not measured — do not quote this repo for any of these:
- No perplexity, KL divergence, or task accuracy. The source ships MMLU artifacts; these
quantized checkpoints were not re-scored against them. A quantization that preserves fluency can
still be worse on tasks, and nothing here rules that out. - No throughput or latency. INT8 W8A8 speed depends on the serving kernel, not the artifact.
- No ablation-robustness test. These models are refusal-removed; quantization may partially
restore refusal. Untested here.
Provenance
| Item | Value |
|---|---|
| Source | distributedcognition/Hemmingway-1-27B-abliterated (BF16 safetensors) |
| Base models | Altworld/Hemmingway-1 and Qwen/Qwen3.8-27B |
| Tool | llmcompressor 0.14.0 / compressed-tensors 0.19.0, model_free_ptq (no calibration set) |
| Method | FP8 weight-only dynamic, weights E4M3 channel-wise symmetric, activations E4M3 dynamic per-token |
| Date | 2026-10-17 |
model_free_ptq uses no calibration dataset: weight scales come from the weights themselves and
activations are quantized at runtime. The artifact is therefore dataset-independent.
License and obligations
Upstream is apache-2.0. That is this artifact's license, but the base models
(Altworld/Hemmingway-1 and Qwen/Qwen3.8-27B) carry their own terms, including regional access limits on the Qwen line. If you
redistribute or serve this checkpoint, you inherit those obligations; read them before deploying.