license: apache-2.0
base_model: philbert440/Qwen3.8-27B-Uncensored-Cyber
tags:

• gguf
• imatrix
• quantized
• agentic
• qwen3
• llama.cpp
• v100
• uncensored
• abliterated
• cyber
  pipeline_tag: image-text-to-text

Qwen3.8-27B-Uncensored-Cyber — GGUF quantized with an agentic imatrix

GGUF quantizations of philbert440/Qwen3.8-27B-Uncensored-Cyber,
calibrated on an importance matrix built from real agentic-coding traffic rather than
generic English prose.

All model weights and behaviour are philbert440's work. This repository contributes
only the quantization and the calibration methodology — no fine-tuning, no merging, no
behavioural change of any kind was performed here.

Inherited from upstream, and stated plainly so nobody is surprised: this is a
de-refusal ("abliterated") model specialized for the cyber/offensive-security domain.
The vision tower and the MTP speculative-decoding head are preserved upstream, which is
what makes the multimodal path and the MTP graft work at all. Read
the base model's card
before using it — it documents the recipe and the evaluation.

Status (2026-08-20): the imatrix and both IQ4_XS artifacts are uploaded.
Sizes and SHA-256 values below are the Hugging Face LFS objects, checked against the
local production artifact for the MTP build.

Files

The two model files have identical IQ4_XS/imatrix text weights. The plus-mtp file adds
the MTP tensors from the matching upstream model; use it only with a runtime that
recognizes that graft. For a conventional GGUF loader, use the non-MTP file unless its
MTP support has been verified. The Q5_K_M comparison below was measured locally and is
not published in this repository.

Why a different calibration corpus

The usual llama.cpp calibration sets (wiki.train.raw, groups_merged.txt) are generic
prose. The tokens that actually break under quantization in agentic use are a different
population:

• tool-call JSON and tool names (bash, read, web_search)
• chat-template special tokens (<|im_start|>, <tool_call>)
• long literals that must be reproduced verbatim — repository paths, package names,
  commit hashes, device UUIDs
• mixed Chinese/English (prose in Chinese, paths and code in English)

Those barely occur in generic prose, so a prose-derived imatrix treats their channels as
unimportant and the quantizer spends its error budget there first. The concrete failure
this targets: an agent writing /home/eze/Documents/PotouI for
/home/ezra/Documents/Proto-UI, then reading its own mangled output back and concluding
its context is inconsistent.

Calibration corpus

Extracted from real agent session logs (an agent CLI driving coding tasks against this
model) — not synthetic prompts:

The corpus itself is not published. It is private working data — real repository
paths, real shell commands, real conversations. The resulting imatrix-agentic-v2.gguf is
published so the method is reproducible; rebuild your own corpus from your own traffic to
reproduce it end to end.

How it was made

# importance matrix, computed on the Q8_0 release
llama-imatrix -m Qwen3.8-27B-Uncensored-Cyber-Q8_0.gguf \
              -f calib-agentic.txt \
              -o imatrix-agentic-v2.gguf \
              -ngl 99 -c 512 --parse-special

# quantization (IQ4_XS shown; Q5_K_M identical apart from the type)
llama-quantize --allow-requantize --imatrix imatrix-agentic-v2.gguf \
               --output-tensor-type q8_0 --token-embedding-type q8_0 \
               Qwen3.8-27B-Uncensored-Cyber-Q8_0.gguf out.gguf IQ4_XS

Three choices worth explaining:

--parse-special is not optional here. The corpus is rendered with the model's own
chat-template markers. Without that flag llama.cpp tokenizes <|im_start|> as literal
text — <, |, im, _start, … — so the special tokens never enter the statistics at
all, which defeats the point of building this corpus. Measured effect on the same corpus
family: 0.342 → 0.299 tokens per byte (−12.7%), consistent with those markers collapsing
into single tokens.

Context stayed at 512, deliberately. The intuitive move is to match the 262 K context
this model is served at. Community testing says the opposite: for a fixed token budget a
small context usually beats 4096, because it yields more and more-diverse samples and a
better-conditioned estimate. 512 is also llama.cpp's long-standing default.

--output-tensor-type q8_0 --token-embedding-type q8_0. Output-layer quantization
error turns directly into picking the wrong token, which is precisely the verbatim-copy
failure above. Costs roughly 1.5 GB.

Honest caveats

• Quantized from the Q8_0 release, not from BF16 (--allow-requantize). Q8_0 is
  near-lossless and the error is dominated by the Q5/IQ4 step, but this is still a second
  quantization.
• Comparison scope is narrow. The measured numbers below use a held-out agentic
  corpus drawn from different sessions than calibration (verified disjoint). They test
  this deployment shape, not general writing, multilingual knowledge, or safety.
• imatrix helps most below Q5_K_M. Community consensus puts the large gains in the
  Q3/Q4 range. IQ4_XS is squarely in useful territory; Q5_K_M is near the upper edge
  where the effect gets modest.
• Calibration is domain-matched to agentic coding. For creative writing or general
  chat, a generic imatrix may suit you better.

Context

Built while making this model usable for agentic work on a single V100 32 GB (SM70) —
FP16-only tensor cores, no INT8/FP8/FP4 TC, no cp.async. On that hardware decode is
memory-bandwidth bound, so IQ4_XS (~4.25 bpw) reads about 26% fewer bytes per token than
Q5_K_M (~5.33 bpw) and frees roughly 4 GB for KV cache — which is why it is the primary
target here rather than the largest quant that fits.

The production configuration used for the measurements below is FastLLM on one V100:
IQ4_XS-imatrix-fromq8-plus-mtp, turbo3 KV cache, batch 4, and a 262,144-token
limit. Multimodal use also requires the matching Qwen3.8 vision projector; it is a
separate artifact and is not embedded in either text-model GGUF here.

Lineage

Qwen3.8-27B                                      — Qwen team
  └─ philbert440/Qwen3.8-27B-Uncensored-Aggressive   α=1.15, recipe-v2 refusal ablation
       └─ philbert440/Qwen3.8-27B-Uncensored-Cyber   residual-cyber peel, β=1.0, apply_from=4
            └─ this repository                       quantization only

Quantized from philbert440's own Qwen3.8-27B-Uncensored-Cyber-Q8_0.gguf release.

Credits

• philbert440 — the Cyber model, its Aggressive
  base, and the Q8_0 GGUF these quantizations are derived from. Everything this repo is
  useful for is downstream of that work.
• Qwen team — Qwen3.8-27B, the underlying model.
• ggml-org/llama.cpp — llama-imatrix and
  llama-quantize. The importance-matrix approach itself comes from
  PR #4861; the GGUF imatrix format
  from PR #9400.
• The llama.cpp community discussions on calibration data and context size
  (#5006), which are why the
  calibration context here stayed at 512 instead of being naively enlarged.

License

apache-2.0, inherited from the base model. These are quantizations of philbert440's
weights; the upstream terms govern.

实测数据(2026-08-20)

质量: 留出集配对困惑度

留出集与校准语料的 512 字节窗口重叠率 0.00% / 0.11%,是干净评测集。
三组用同一份留出集、同一个 --chunks 60。

不要按这两个置信区间去比。 两组评的是同一批分块,误差高度相关,
按独立区间读会严重低估判别力。做配对比较(逐块还原 NLL 后对差值做检验):

IQ4_XS vs Q5_K_M   ΔPPL = +0.417% ± 0.187%   t = 2.22   60 块中 39 块更差

也就是说:从 5.33 bpw 降到 4.25 bpw 的代价是 0.42%,差异真实、方向一致,
但幅度很小。配对把不确定度从 ±1.74% 压到 ±0.187%,约 9 倍。

功能: 走真实推理栈(fastllm on V100)的验收

"逐字抄写"测的是模型把上下文里的路径、UUID、commit 哈希原样复制的能力。
它比困惑度更贴近 agent 的真实失败模式:抄歪一个字符命令就失败,而困惑度
是全词表平均,对这种尾部行为几乎不敏感。

速度: 控制上下文长度后的配对结果

早期把不同上下文长度的请求直接比较,一度误判 IQ4_XS 比 Q5_K_M 慢。控制成
相同 token 形状后,8 组端到端配对样本全部同向:IQ4_XS 平均为 Q5_K_M 的
1.20x。这是单卡 V100 + FastLLM 的部署结果,不是跨推理栈的通用保证。

当前生产仍有明确的算子余量:decode 的主路径是通用 mmvq,而 SM70 IQ4_XS
MMQ 只覆盖较大的 n;因此 1.20x 主要来自更低的权重带宽和当前融合路径,
不代表专用 decode kernel 已经用满硬件。