license: apache-2.0
base_model:

• jakeroxs/KAT-Coder-V2.5-Dev-35B-A3B-MTP-ABLITERATED-GGUF
• Kwaipilot/KAT-Coder-V2.5-Dev
  base_model_relation: quantized
  tags:
• gguf
• rocm
• vulkan
• rocmfp4
• strix-halo
• gfx1151
• amd
• moe
• mtp
• speculative-decoding
• code
• abliterated
• uncensored
  pipeline_tag: text-generation

KAT-Coder-V2.5-Dev-Abliterated-MTP — ROCmFP4 for AMD Strix Halo (gfx1151)

The model here is not our work. The abliterated MTP checkpoint is
jakeroxs's
(KAT-Coder-V2.5-Dev-35B-A3B-MTP-ABLITERATED-GGUF);
the base model is Kwaipilot's
(KAT-Coder-V2.5-Dev). This repository adds
only the ROCmFP4/ROCmFPX quantisation ladder, the dual-backend build, and the measurements.
Go star their repos.

Four quantisations of the abliterated KAT-Coder-V2.5-Dev 35B-A3B MoE, each carrying a working
Multi-Token-Prediction block, built for AMD Ryzen AI Max+ 395 / Radeon 8060S. Every file runs on
BOTH the HIP (ROCm) and Vulkan backends from a single binary — the backend is a runtime -dev
flag, not a rebuild.

Lead result: MTP is worth +6.5 % on Vulkan and +8.9 % on ROCm here, and the backend
crossover flips by tier — Vulkan takes the 4-bit tiers, ROCm takes both Q6 tiers and runs
them in 7.2 GiB less GTT.

✅ This model answers cleanly out of the box. 6/6 non-empty content at every budget from 400
tokens up, on all four workloads. That is worth stating because its architectural sibling does
not — see the note at the end.

Which file should I use?

STRIX_LEAN (106) on Vulkan is the pick for coding work — fastest on the code workload
(80.76 tok/s) and the smallest file. If your work is prompt-heavy, the same file on -dev ROCm0
prefills at 478 tok/s.

⛔ ftype 105 (STRIX) is deliberately not published — it leaves output.weight and the
*_shexp tensors at 4 bit.

Quick start — the backend is one flag

# Vulkan — best decode (chat, agentic loops)
llama-server -m KAT-Coder-V2.5-Dev-Abliterated-MTP-Q4_0_ROCMFP4_STRIX_LEAN.gguf \
  -dev Vulkan0 -ngl 999 -fa on -fit off -np 1 -c 8192 --jinja \
  --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.0 \
  --host 0.0.0.0 --port 8080

# HIP / ROCm — best prefill (long prompts, cold starts, RAG)
#   SAME FILE. Only -dev changes.
llama-server -m KAT-Coder-V2.5-Dev-Abliterated-MTP-Q4_0_ROCMFP4_STRIX_LEAN.gguf \
  -dev ROCm0 -ngl 999 -fa on -fit off -np 1 -c 8192 --jinja \
  --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.0 \
  --host 0.0.0.0 --port 8080

--spec-draft-p-min 0.0 is not optional if you want the speed. The default is 0.75 and it
throttles the draft.

⛔ You need a ROCmFPX build with Vulkan enabled — stock llama.cpp will NOT load these files

ROCmFP4/ROCmFPX use ggml tensor types 100–119. Upstream llama.cpp's type table stops at 43, so a
stock build rejects these files outright, on any backend:

gguf_init_from_reader: tensor '...' has invalid ggml type 101. should be in [0, 43)

Build one with both backends — the exact configuration behind every number on this card:

git clone https://github.com/charlie12345/ROCmFPX.git && cd ROCmFPX
cmake -B build-hipvk -G "Unix Makefiles" \
  -DCMAKE_MAKE_PROGRAM=/usr/bin/gmake \
  -DCMAKE_C_COMPILER=/usr/bin/cc -DCMAKE_CXX_COMPILER=/usr/bin/c++ \
  -DCMAKE_HIP_COMPILER=/opt/rocm-7.2.4/lib/llvm/bin/clang \
  -DROCM_PATH=/opt/rocm-7.2.4 -DCMAKE_PREFIX_PATH=/opt/rocm-7.2.4 \
  -DGGML_HIP=ON -DGGML_VULKAN=ON -DAMDGPU_TARGETS=gfx1151 \
  -DVulkan_GLSLC_EXECUTABLE=/usr/bin/glslc \
  -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF -DLLAMA_BUILD_WEBUI=OFF
cmake --build build-hipvk -j12
build-hipvk/bin/llama-server --list-devices   # must list ROCm0 AND Vulkan0

Build traps, all hit for real: no Ninja on this box — use Unix Makefiles + gmake, and pass
-DCMAKE_C_COMPILER/-DCMAKE_CXX_COMPILER explicitly or the HIP language probe dies with
CMAKE_C_COMPILER not set, after EnableLanguage; the HIP compiler is at
/opt/rocm-7.2.4/lib/llvm/bin/clang, not /opt/rocm/llvm/bin/clang++; glslc is not
auto-detected.

Measured performance

AMD Ryzen AI Max+ 395 / Radeon 8060S (gfx1151), 128 GB unified, GTT 131072 MiB, ROCm 7.2.4,
Mesa 25.2.8 (RADV), kernel 6.17.6, Ubuntu 24.04.4, governor performance, box idle — 0
co-resident seats. -c 4096 -fa on -fit off -np 1, temperature 0, cache_prompt:false, unique
nonce per request, 400 tokens, 1 warm-up discarded, median of 12.

All four tiers × both backends (MTP on, p_min 0.0)

What MTP is worth — STRIX_LEAN, same file, speculation off vs on

With enable_thinking: false, 1600 tokens

The findings that matter

1. The backend crossover flips by tier. Vulkan wins decode on the 4-bit tiers; on both Q6
   tiers ROCm wins outright — Q6_LEAN 64.89 vs 59.08 — and uses 7.2 GiB less GTT (28.1 vs
   35.3). ROCm keeps the prefill crown everywhere (478 vs 260 on STRIX_LEAN, ~+84 %).
2. MTP pays on both backends (+6.5 % / +8.9 %), acceptance 3.01–3.20 of 4 on every arm.
3. Q6 buys footprint, not speed. Q6_AGENT (30.89 GiB) runs slower than STRIX_LEAN
   (17.88 GiB) on every workload. Choose Q6 for precision, not throughput.

⚠️ These rankings are model-specific and we say so. Measure them on your own model.

Silent-truncation gate — 480 generations, 0 truncated

On this hardware a Vulkan failure can be silent: 1 token, empty content, clean exit, /health
still 200. A benchmark reporting only a median cannot see it. We count generations that actually
reach n_predict:

Speculative decoding works on Vulkan on this model, across 240 Vulkan generations.

Content adequacy — clean at every budget

Non-empty content, 6 reps per cell, thinking at default:

We measured this because the architectural sibling we built alongside it — an abliterated
Agents-A1 on the same qwen35moe 35B-A3B graph, same fork, same box — returns empty content on
most non-code prompts at every budget up to 1600 unless enable_thinking:false is set. This model
has no such problem. Same architecture, opposite behaviour: do not assume a serving parameter
transfers between models on this graph.

Head protection — verified, not assumed

Left to their defaults several ftypes emit a 4-bit output.weight, a documented
quality-collapse pattern. Every file here was quantised with an explicit head flag and then read
back out of the finished file by exact tensor name — never substring-matched, because
attn_output matches output and gives a false pass.

On 106 token_embd is deliberately not overridden — Q5_K embeddings are the point of the LEAN
tier; only the head needs forcing.

Source integrity

The source checkpoint was gated before quantisation, not assumed:

arch                 qwen35moe
tensors              753   (blk.0-39 + blk.40 MTP)
block_count          41
nextn_predict_layers 1
blk.40 tensors       20
output.weight        BF16   (unquantised source)
bytes                71,066,994,848

Quantised from BF16 only — never from an existing Q4_K_M or ROCmFP4.

Reproduction block

binary          llama-server  sha256 d077b98f3f04df4b482a5826c8922e39cda3aa8d2e2b2fbe1870f36d245cebe7
                llama-quantize sha256 1559189968a30781f837e4d25ca3cdf26c8eb0351023675406535eddcbe0db13
source          ROCmFPX @ d3ca53726109b46fd5cd6fa42f688c707079b44e  (-DGGML_HIP=ON -DGGML_VULKAN=ON)
hardware        AMD Ryzen AI Max+ 395 w/ Radeon 8060S (gfx1151), 128 GB unified
gtt_total       131072 MiB
rocm            7.2.4          vulkan  Mesa 25.2.8 (RADV GFX1151), API 1.4.318
kernel          6.17.6-061706-generic      os  Ubuntu 24.04.4 LTS
governor        performance
co-resident     0 seats (box idle for the whole measurement)
env             HSA_OVERRIDE_GFX_VERSION=11.5.1
serve argv      -ngl 999 -fa on -fit off -np 1 -c 4096 --jinja
                --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.0
                -dev {ROCm0|Vulkan0}
quantise argv   llama-quantize <headflag> <BF16> <OUT> <FTYPE> 16
protocol        temperature 0, cache_prompt false, unique nonce, 400 tok,
                1 warm-up discarded, median of 12

SHA256

3a4e59d1f7dfb8257ba81604f5739fa556de02f18887205f136f8fa427c8cd4f  KAT-Coder-V2.5-Dev-Abliterated-MTP-Q4_0_ROCMFP4_COHERENT.gguf
21278799c27afa5894c7626fdf6b3717b0a30fa7f8fa4b63aa831bb8e5904d01  KAT-Coder-V2.5-Dev-Abliterated-MTP-Q4_0_ROCMFP4_STRIX_LEAN.gguf
a016cd522b15666b719bc087b03f07234cc140fc663449d37f1e9dc59b4cedae  KAT-Coder-V2.5-Dev-Abliterated-MTP-Q6_0_ROCMFPX_AGENT.gguf
f92370a5b2b513b940bf46fb53c9e26898e5d85c91f84c8d85ff3bfe8bc4dfbf  KAT-Coder-V2.5-Dev-Abliterated-MTP-Q6_0_ROCMFPX_LEAN.gguf