license: apache-2.0
language:
- ru
- en
pipeline_tag: text-generation
base_model: - huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated
base_model_relation: quantized
datasets: - HuggingFaceFW/fineweb-2
- HuggingFaceFW/fineweb-edu
- Vikhrmodels/GrandMaster-PRO-MAX
- open-r1/Mixture-of-Thoughts
- lightblue/reasoning-multilingual-R1-Llama-70B-train
- nvidia/OpenCodeReasoning
- ise-uiuc/Magicoder-OSS-Instruct-75K
tags: - gguf
- qwen35moe
- ternary
- tquant
- mtp
- ptq1-0-g64
- prism
- abliterated
- experimental
Huihui-Qwen3.6-35B-A3B · TQuant GGUF
FesarovLab · Russian & English · Text only · Ternary MoE · Optional MTP
| Target model | Expert storage | Target routed layers | Optional MTP file |
|---|---|---|---|
| 9.38 GB | 1.875 bits/weight | 40 / 40 ternary | 9.88 GB |
Downloads · Quick start ·
Method · Evaluation ·
Runtime & build · Technical report
Experimental text-only compression of
Huihui's abliterated Qwen3.6-35B-A3B.
All 40 target-model routed-expert layers use ternary codes with trained group-64 scales.
The other components use higher precision. This is not a fully ternary model.
Requires the included, modified Prism runtime. Stock llama.cpp and an unmodified Prism release
are not supported for these files. GGUF tensor type 144 (PTQ1_0_G64) and Hadamard metadata
must both be understood by the runtime. See RUNTIME.md.
No Transformers, Ollama, LM Studio, Metal, Vulkan or hosted Inference Provider compatibility is claimed.
This release does not reach Q4_K_M quality on every category: web and Russian assistant responses
remain weaker. See the evaluation tables below.
Files
| File | Bytes | Decimal GB | Contents |
|---|---|---|---|
| T40g64-fd-old256.gguf | 9,379,655,776 | 9.380 | Recommended baseline; no draft head |
| T40g64-fd-old256-mtp.gguf | 9,876,012,448 | 9.876 | Same target + optional MTP draft block |
The MTP variant adds 496,356,672 bytes. Its extra draft block has Q4_K routed experts;
the main model's 40 routed layers remain ternary. Do not describe the MTP file as having exclusively
ternary expert layers. Each file is self-contained; download only the desired variant.
Hashes and tensor inventories are in release-manifest.json.
Quick start (Windows / NVIDIA)
Download one variant (authenticate first while this repository is private):
hf download FesarovLab/Huihui-Qwen3.6-35B-A3B-abliterated-TQuant-GGUF qwen36_35b_v1-T40g64-fd-old256.gguf --local-dir .
Build the pinned runtime from runtime/prism-tquant-source.zip as described in RUNTIME.md.
The examples assume its Release binaries are on PATH. Use the embedded Jinja template.
RTX 4070 Laptop, 8 GB VRAM, 64 GB system RAM (experts on CPU):
llama-server.exe -m qwen36_35b_v1-T40g64-fd-old256.gguf -ngl 99 --n-cpu-moe 40 -t 12 -c 8192 --jinja --host 127.0.0.1 --port 8080
RTX 5070 Ti, 16 GB VRAM, 32 GB system RAM (try full GPU offload):
llama-server.exe -m qwen36_35b_v1-T40g64-fd-old256.gguf -ngl 99 -t 8 -c 8192 --jinja --host 127.0.0.1 --port 8080
The 16 GB full-offload benchmark used an earlier checkpoint with the same format and size;
the final old256 server at 8K context still requires a local memory check. Model file size is not
total RAM/VRAM use. KV cache, compute buffers and concurrent requests need additional memory.
Start with one request. No maximum-context or concurrency capacity is promised.
For MTP, select the -mtp.gguf file and append:
--spec-type draft-mtp --spec-draft-n-max 1
On the laptop, 2–3 draft tokens were slower. Greedy text sometimes differs from non-speculative
decoding; numerical differences during batched verification are a possible cause, not a proven
explanation of every divergence. Exact text equivalence has not been established.
To disable MTP, omit the speculative flags; prefer the smaller base file if it is never needed.
Example OpenAI-compatible request, with thinking disabled:
curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"messages":[{"role":"user","content":"Explain how a hash table works."}],"temperature":0,"max_tokens":256,"chat_template_kwargs":{"enable_thinking":false}}'
This is HTTP access to a local Prism server, not an official OpenAI-hosted model.
Thinking can be enabled through the chat-template parameter; thinking tokens count toward the limit.
Compression and provenance
- Architecture:
qwen35moe, 40 target layers, hidden size 2048, 256 routed experts / top-8 per layer,
plus gated shared experts; attention interleaved with Gated DeltaNet. Text only; no vision projector. - Target expert format:
PTQ1_0_G64, 128 weights in 30 bytes (26 bytes of base-3 trits and two FP16
scales), 1.875 storage bits per expert weight, not the whole-model average. - Block-512 signed Hadamard folding, retained as
prism.hadamard.*GGUF metadata. - Non-expert recipe: attention Q6_K; DeltaNet output Q8_0, remaining selected DeltaNet weights Q4_K;
shared experts Q6_K; token embeddings Q4_K; output Q6_K. Norm/router exceptions remain higher precision. - Expert scales trained against the BF16 teacher with top-64 KL + 0.1 CE on local RU/EN mixtures,
2048-token windows; code/chat/thinking use assistant masks, web uses all targets. - Selected run:
i04_fd_old256, best step 224 of a 256-update full-depth continuation from late128.
Lineage: e2e4b → E0g64 → restart128 → clean512 → late128 → fd-old256. - This published checkpoint has frozen ternary codes. Later GPTQ experiments are not included.
- MTP weights come from the same source teacher, without separate draft training.
Source revisions observed during release preparation are recorded in the manifest. They are not
asserted to be the original download revisions where that evidence is unavailable.
Data provenance and licensing notes: DATA_PROVENANCE.md.
Technical report
The Russian-language technical manuscript (PDF) and
editable Markdown describe the Hadamard transform, exact
base-3 byte layout, learned scales, distillation loss, GGUF export and runtime integration.
It also records negative experiments and reproducibility limits. This is a working
technical report, not a peer-reviewed publication.
Evaluation
Exact-target runtime NLL on matching token manifests, candidate minus Q4_K_M;
negative is better. Cells with a slash are all tokens / assistant tokens.
These are local evaluation categories, not public benchmark leaderboard scores.
| Model | Bytes | code | ru_chat | en_web | ru_web | think_en | think_multi |
|---|---|---|---|---|---|---|---|
| T40g64 fd-old256 | 9,379,655,776 | −0.043 / −0.010 | −0.205 / +0.071 | +0.174 | +0.143 | +0.008 / +0.005 | −0.039 / +0.025 |
| Q4_K_M reference | 21,166,758,560 | 0 | 0 | 0 | 0 | 0 | 0 |
Window counts (all): code 32 / ru_chat 32 / en_web 32 / ru_web 22 / think_en 32 / think_multi 4.
The multilingual thinking sample is particularly small. Chat quality should be judged using assistant
tokens; lower all-token chat NLL is not proof of better conversation quality.
Packing-boundary targets affect these official scores; they should not be interpreted as isolated-document
likelihoods. No broad quality-equivalence claim follows from a 12-prompt generation sanity check.
Full-file perplexity, context 2048, special-token parsing enabled:
| Model | Bytes | code | ru_chat | en_web | ru_web | think_en | think_multi |
|---|---|---|---|---|---|---|---|
| T40g64 fd-old256 | 9,379,655,776 | 2.058 | 3.133 | 8.135 | 10.955 | 1.912 | 1.686 |
| Q4_K_M | 21,166,758,560 | 2.143 | 3.685 | 6.933 | 9.535 | 1.902 | 1.683 |
No separate MTP perplexity run is claimed. Its target tensors are checked byte-for-byte against the
base file; MTP is a decoding option. Generation parity is a separate, unresolved numerical question.
Speed (limited measurements)
Final old256 MTP-file server on RTX 4070 Laptop / 8 GB, experts on CPU, -t 12 -c 8192, warm file cache,
thinking off, 256 generated tokens, three prompts (2026-11-05):
| Configuration | Russian chat | English code | English prose |
|---|---|---|---|
| No speculation, two runs | 30.7 / 30.9 | 28.5 / 28.0 | 27.9 / 29.2 |
| MTP, one draft token | 33.7 | 31.0 | 28.4 |
Units: generated tokens/second. Different output text and few repetitions limit comparability.
The initial cold-cache run is excluded. Do not interpret this as a guaranteed speedup.
An earlier same-size T40g64 checkpoint on RTX 5070 Ti / 16 GB achieved tg128 162.1 ± 1.0 tokens/s
in llama-bench, with full offload; this is neither a final-old256 nor an MTP benchmark.
Limitations and attribution
The source model was abliterated by huihui-ai; reduced refusal behavior is inherited.
Incorrect and inappropriate outputs remain possible. Web quality is materially below Q4_K_M.
Tool calling, long-context behavior, production serving and all GPU architectures have not been certified.
Base-model license: Apache-2.0, attributed to Qwen and huihui-ai.
Runtime source: MIT, with its upstream licenses retained inside the source archive and in runtime/LICENSE.
See NOTICE and DATA_PROVENANCE.md for modification and dataset attribution.
This is an independent T_Quant derivative, not an official Qwen, huihui-ai or PrismML release.