license: apache-2.0
library_name: llama.cpp
pipeline_tag: image-text-to-text
base_model:

• Qwen/Qwen3.8-27B
• JonathanColetti/Qwen3.8-27B-Uncensored
• mradermacher/Qwen3.8-27B-Uncensored-i1-GGUF
• incoai/Qwen3.8-27B-DFlash2
  language:
• en
• zh
  tags:
• gguf
• qwen3.8
• qwen35
• uncensored
• abliterated
• speculative-decoding
• dflash2
• loopspec
• pascal
• sm_60
• gp100
• vision

Qwen3.8-27B Uncensored — GP100 pack

Runtime: thefallentree/llama.cpp-gp100 (pr-sm60). That fork is the only public llama.cpp with the sm_60 HFMA2 mat-vec, Q4_1_G64, the tensor-split meta-backend fixes, and LoopSpec (ngram-mod,draft-dflash) that these numbers were measured on. Stock llama.cpp will load the Q4_1 files and run them slowly. It will not load Q4_1_G64.

This is not a new train. It is the Q4_1 / Q8_0-head pack that actually crossed 100 tok/s on 2× Tesla P100, plus the DFlash2 draft and F16 mmproj that LoopSpec uses.

What reached 100+ tok/s

These Q4_1 files, not G64.

Canonical /completion bench on 2× P100, 256 tokens, 5 reps, warm = median of runs 2–5, tensor-split 1,1, Q8 KV, LoopSpec ngram-mod,draft-dflash. Same completion hash e1d5bc16.

The 106–108 figure is warm exact ngram reuse (accept ~0.99). Cold / novel text falls back to DFlash2 every round and sits at ~55–61. Do not quote 107 as universal decode speed.

First crossing of 100 (2026-08-30): warm reuse 102.7 tok/s (min 100.7) across 12 prompt categories, 512 tokens, accept 0.987–0.992.

Run

git clone https://github.com/thefallentree/llama.cpp-gp100
cd llama.cpp-gp100
git checkout pr-sm60
# CUDA build, same flags you would use for llama-server

./build/bin/llama-server \
  --model Qwen3.8-27B-Uncensored.i1-Q4_1-Q8_0-head.gguf \
  --mmproj mmproj-Qwen3.8-27B-Uncensored-F16.gguf \
  --spec-type ngram-mod,draft-dflash \
  --spec-draft-model Qwen3.8-27B-Uncensored-DFlash2-Q4_1-private-q4-head-exact-embd.gguf \
  --spec-draft-n-max 3 \
  --spec-ngram-mod-n-max 7 \
  --device CUDA0,CUDA1 --split-mode tensor --tensor-split 1,1 \
  --device-draft CUDA1 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --ctx-size 131072 --flash-attn on

--backend-sampling is on by default in that fork's pr images (GPU greedy verify). Leave it off on a tensor-split Gemma; it is fine on this Qwen pack.

Why Q4_1 (and why G64 next)

GP100 has no DP4A and no tensor cores. The fork's fast path (ggml_cuda_mmvq_f16_sm60) reads Q4_1 / Q4_1_G64 / Q4_K / Q8_0. Q4_1's 20-byte block is always 4-byte aligned. Q4_0 is rejected. Community Q4_K_M names are not evidence of that layout — a file labeled Q4_K_M on this disk (Hauhau Gemma) is IQ2_XXS / Q6_K / Q8_0 / Q2_K with zero Q4_K.

This Qwen pack's name is true: every quantized 2-D weight is Q4_1 or the one Q8_0 head. Header dump: 866 tensors, no type/shape drift. See TENSORS.md.

Q4_1_G64 is the same nibbles with one (d,m) per 64 weights (4.5 bpw). Kernel-only vs this Q4_1: +12% width 1, +7% width 4, +6% width 8, PPL +1.0%. It does not create the 107 number — LoopSpec already skips most mat-vecs on warm reuse. It does pay on cold verify, target-only, and VRAM (it was the only pack that fit 256K × 8 slots without a drafter). Needs llama.cpp-gp100 (type id 43).

sm_60 only. P40 / GTX 10-series are sm_61; the HFMA2 path is gated off there.

Tensors (target)

17.652 GB, arch qwen35, 65 blocks (48 gated-attn+SSM, 17 full-attn), vocab 248320, native MTP on blk.64.

506 / 554 tensors with ndims>=2 pass the HFMA2 weight-side checks. The 48 misses are F32 ssm_conv1d. Dense; no expert ids.

Provenance

License

Apache 2.0, inherited from Qwen/Qwen3.8-27B. Local inference and research. Not a safety-tuned deployment model. Abliteration reduces refusals; it does not remove them.