base_model: AEON-7/Qwen3.8-27B-AEON-Ultimate-Uncensored-BF16
library_name: transformers
pipeline_tag: image-text-to-text
tags:

• vllm
• compressed-tensors
• autoround
• quantized
• w4a16
• intel-xpu
• multimodal

Qwen3.8-27B AEON Ultimate Uncensored — W4A16 AutoRound

This repository contains a W4A16 quantization of
AEON-7/Qwen3.8-27B-AEON-Ultimate-Uncensored-BF16,
exported as compressed-tensors for vLLM. It also contains experimental
dataset-calibrated static FP8 KV-cache scales; testing found those scales unsafe
for serving this revision, as documented below.

The quantized language model is combined with the exact BF16 vision tower and
BF16 MTP tensors from the pinned source revision. These tensors were grafted
without conversion after quantization because Transformers loads this model's
text backbone separately from its conditional-generation wrapper.

The quantization was produced for Intel XPU serving. Other hardware and serving
engines may work if they support this compressed-tensors scheme, but have not
been validated for this release.

Reproducibility

The source revision was resolved and pinned before calibration. Weight
quantization and FP8 KV-scale calibration used the same aligned calibration
dataset.

Full-precision exclusions

The following model-specific selectors were excluded from AutoRound weight
quantization:

• lm_head
• token embeddings (re:.*embed_tokens$; embeddings are not Linear targets)
• DeltaNet/linear-attention A and B input projections
  (re:.*linear_attn\.in_proj_[ab]$)

The exported config.json contains the fully resolved compressed-tensors ignore
list. These exclusions apply to weight quantization. KV-cache calibration is a
separate modifier and remains represented by kv_cache_scheme plus
model-kv-scales.safetensors.

Serving with vLLM

Use the backend's native cache dtype. This configuration has been tested on
Intel XPU:

vllm serve jasonboukheir/Qwen3.8-27B-AEON-Ultimate-Uncensored-BF16-W4A16-AutoRound \
  --quantization compressed-tensors \
  --kv-cache-dtype auto \
  --max-model-len 65536

For this BF16 model, auto selects the backend's native BF16 KV cache. Do not
serve this revision with --kv-cache-dtype fp8. A long-generation test using
the included calibrated static scales degenerated into severe token and phrase
repetition. Repeating the test with the same quantized weights and a BF16 KV
cache produced coherent output. This isolates the observed failure to the FP8
cache path or its scales rather than demonstrating a W4A16 weight failure.

The FP8 metadata and scale shard remain in the repository for reproducibility
and diagnosis, not as a serving recommendation. They need recalibration plus
layerwise and long-context parity evaluation before they should be enabled.

The source architecture combines linear-attention and periodic full-attention
layers. Confirm that your vLLM version supports this Qwen architecture and the
chosen XPU kernels.

The restored Qwen3_5ForConditionalGeneration wrapper supports image inputs as
well as text. Image preprocessing and OpenAI-compatible request syntax depend
on the serving version; consult the matching vLLM multimodal documentation.

Validation status

Completed checks:

• the quantization run completed successfully;
• the output was serialized as compressed-tensors;
• W4A16 metadata is present in config.json;
• static FP8 KV metadata and model-kv-scales.safetensors are present;
• the published artifact manifest records SHA-256 hashes for every file;
• the BF16 MTP and vision tensors were restored from the pinned source and the
  conditional-generation checkpoint loaded successfully;
• vLLM selected its Intel XPU W4A16 compressed-tensors kernel;
• MTP drafting, compiled XPU graphs, multimodal warmup, and a 65,536-token
  serving window initialized successfully;
• generation with a BF16 KV cache was coherent;
• generation with the included calibrated FP8 KV scales exhibited severe
  repetitive degeneration and is not recommended;
• the complete grafted checkpoint is approximately 18.12 GiB.

Not yet reported for this release:

• perplexity or downstream quality evaluation;
• generation parity against the BF16 source;
• systematic long-context and layerwise KV-cache sensitivity evaluation;
• throughput, latency, or memory benchmarks.

No quality or speed improvement over the source model is claimed until those
tests are published.

Intended use and limitations

This is a quantized derivative of the source model. Its behavior, intended use,
training history, safety characteristics, and content limitations originate
with the source model. Quantization may introduce additional quality loss or
behavior changes.

The source is described as uncensored. It may generate inaccurate, harmful,
offensive, or otherwise objectionable content. Do not treat its output as
professional, legal, medical, financial, or safety-critical advice. Operators
are responsible for appropriate evaluation, access controls, and safeguards.

License

No independent license is asserted by this quantization repository. Review and
comply with the license and terms published by the
AEON-7/Qwen3.8-27B-AEON-Ultimate-Uncensored-BF16
source repository and any upstream base models before using or redistributing
these weights.

Quantization tooling

Created with the reproducible workflow in
jasonbk/vllm-xpu-nix,
using llm-compressor's AutoRound modifier and a pinned Intel XPU toolchain.