license: mit
base_model: dealignai/GLM-5.3-Flash-UNCENSORED-FP8
base_model_relation: quantized
tags:
- exl3
- quantized
- glm
- glm-5.3-flash
- tensorfold
- dgx-spark
- uncensored
library_name: transformers
GLM-5.3-Flash-Uncensored — EXL3 3.0 bpw (mcg), TensorFold / 2× DGX Spark
A 3.0-bit EXL3 quantization of dealignai/GLM-5.3-Flash-UNCENSORED-FP8 (MIT), with only the 288 routed experts per MoE layer quantized (EXL3, mcg codebook) and every non-routed tensor kept at source-native precision. Built to run on a 2× NVIDIA DGX Spark (GB10) pair with TensorFold, TP=2 over the direct CX7 link.
Full measured results (quality / KV pool / speed, incl. comparison against Mia-AiLab's official 4bpw): see results/benchmarks.md.
What's inside
| File | Content |
|---|---|
model-0000X-of-00018.safetensors ×18 + model.safetensors.index.json |
Weights, 152 GB total (total_size 152,087,838,592 bytes ≈ 141.6 GiB) |
config.json |
quantization_config: {quant_method: exl3, bits: 3, codebook: mcg, head_bits: 16, scope: glm53_routed_experts_only, non_routed_dtype_policy: official_source_native} |
tokenizer.json / tokenizer_config.json / processor_config.json / chat_template.jinja |
From the base repo, unmodified |
generation_config.json |
From the base repo |
LICENSE |
MIT (© 2026 Z.AI Co., Ltd), inherited from the base |
results/ |
Our benchmark artifacts (EvalPlus raw per-task results + full write-up) |
tools/ |
Quantization / grafting / verification scripts used to produce this pack |
How it was quantized
- Routed experts quantized with upstream exllamav3 v1.6.0 (
convert.py -b 3.0 -cb mcg -hb 16 -eb 16 -mb 16 -vb 16) from the FP8 base. tools/graft_experts_only.pygrafts the EXL3 expert tensors back onto the source-native non-expert tensors (dense MLP, attention, shared experts, embeddings, head, vision tower — all at source precision; FP8 source tensors were repacked to BF16 with theirweight_scale_invscales preserved).config.jsoncarries the graft provenance (grafted_by: tf-glm-exl3-lowbit/tools/graft_experts_only.py).
Quality (measured on our stack)
| Benchmark | This 3.0bpw (measured) | Mia-AiLab 4bpw (self-reported) |
|---|---|---|
| EvalPlus: HumanEval+ & MBPP+ (542 problems, thinking on) | 86.3% (468/542) | 86.5% (469/542) |
| GSM8K (250, greedy) | 98.0% | 98.8% |
| HumanEval (164, greedy) | 98.2% | 95.7% |
All three differences are within noise (Mia's own paired tests on her two 4bpw packs show ±2–3 problem flips with p = 0.45–1.0). Not yet measured for this pack: KL divergence vs the FP8 source, and 1M-token needle recall (tested only up to ~64k). Raw per-task EvalPlus results are in results/evalplus/.
Size, KV pool and speed — ⚠️ read the config note
| Metric | This 3.0bpw | Mia-AiLab official 4bpw |
|---|---|---|
| Pack size | 152 GB | 175.7 GB (−13.5%) |
| KV pool (our tuned serving config, see below) | 3,891,200 tokens | 2,582,528 tokens (+51%) |
| Forward-pass latency, same machine & engine (spec-decode verify rows 1 / 4 / 8) | 24.3 / 38.4 / 49.0 ms | 26.6 / 45.7 / 59.8 ms (avg ≈ +15% faster) |
⚠️ Configuration disclaimer — please read before replicating
Mia's officially claimed KV cache (from her model card): KV pool 2,582,528 tokens at TP=2, 4 streams, 1M-token window, FP8 KV. Her serving-kit README separately states a default of 850k context / 1.23M-token KV pool. Our numbers above were NOT measured under either of those configurations. Our serving stack runs
--tp 2 --parallel 8 --context 1048576with a tuned memory reserve (~26.5 GiB); at a Mia-equivalent reserve this same 3bit pack serves a 2,582,528-token pool (parity), and at our tuned reserve it reaches 3,891,200 tokens.Do not assume Mia's pool/context numbers transfer to this pack. On unified-memory GB10 machines the KV pool is whatever the memory reserve leaves behind: set the reserve too small and you get
NV_ERR_NO_MEMORY(and on DGX Spark that can take the whole box down), set it too large and your pool shrinks. Replicate with the config stated next to each number inresults/benchmarks.md.（中文注记：上表我们的数字是在我们自己的配置下实测的，不是 Mia 官方声称的 258 万 token 池/850k 上下文那套配置——我们本地并没有按她的配置跑。请按
results/benchmarks.md里每项数字旁标注的配置复现；统一内存机器上 reserve 给小了会NV_ERR_NO_MEMORY直接把机器干崩，给大了池子反而变小。）
Usage (TensorFold, 2× DGX Spark)
Requires a TensorFold build with 3-bit EXL3 expert support (the stock gate only accepts bits: 4; see tools/):
tensorfold serve /path/to/this/repo \
--tp 2 --context 1048576 --parallel 8 --max-tokens 32768 \
--thinking --vision --spill-gib 300
Measured live on 2× DGX Spark (GB10, CX7 direct 10.0.0.1/10.0.0.2): KV pool 3,891,200 tokens, engine startup ~111 s.
Attribution and changes
- Base: dealignai/GLM-5.3-Flash-UNCENSORED-FP8 (MIT, © 2026 Z.AI Co., Ltd) — upstream: zai-org/GLM-5.3-Flash (MIT).
- Changes: routed experts requantized EXL3 3.0bpw/mcg; non-routed tensors repacked FP8→BF16 preserving scales; tokenizer/chat template/processor config unchanged from base. The base repos are untouched.
- Not affiliated with dealignai or Z.ai.
- Comparison baseline "Mia-AiLab 4bpw" = Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold numbers as self-reported on her model card, measured on her own hardware/configuration.
- Author: x.com/greatYiTin · github.com/greatyingzi