license: apache-2.0
base_model: huihui-ai/Huihui-Ornith-1.5-35B-A3B-abliterated
base_model_relation: quantized
tags:
- ornith
- abliterated
- uncensored
- w8a8
- int8
- compressed-tensors
- vllm
- multimodal
language: - en
pipeline_tag: image-text-to-text
Ornith-1.5-35B-A3B-abliterated-W8A8
INT8 W8A8 quantisation of
huihui-ai/Huihui-Ornith-1.5-35B-A3B-abliterated,
which is an abliteration of ornith-ai/Ornith-1.5-35B-A3B.
35.7 GiB, compressed-tensors / int-quantized.
Why this one
huihui-ai's abliteration is the only one of the three that keeps the model
whole. This quantisation preserves that:
| here | other published W8A8 | |
|---|---|---|
| architecture | Qwen3_5MoeForConditionalGeneration |
Qwen3_5MoeForCausalLM |
model.visual.* tensors |
333 | 0 |
| MTP head | yes | yes |
| abliterated | yes | no |
The two other abliterations of this model drop both the MTP head and the 333
vision tensors, and one of them also downgrades the architecture to the
text-only ForCausalLM. A text-only checkpoint silently drops every image and
OCR caller, so the vision tower is not optional.
Why W8A8 rather than FP8 or NVFP4
On Ampere (compute capability 8.6, e.g. A40), vLLM's W8A8 INT8 scheme declares a
minimum capability of 75, so the card multiplies in its own INT8 tensor cores.
FP8 W8A8 declares 89, so below Lovelace vLLM falls back to the weight-only
scheme and the activations go back to 16 bits: the memory saving is real and the
speedup is not. NVFP4 carries no native FP4 path on Ampere at all.
Serving
vllm serve nuoram/Ornith-1.5-35B-A3B-abliterated-W8A8 \
--tensor-parallel-size 2 \
--enable-auto-tool-choice --tool-call-parser qwen3_xml
Provenance
Quantised with llm-compressor from the bf16 checkpoint; the recipe is inrecipe.yaml in this repo. The MTP head is carried as model-mtp.safetensors.
Abliteration is by huihui-ai and the base model by ornith-ai. All credit for the
model itself belongs to them. This repo only changes the numeric format.