license: mit
base_model: huihui-ai/Huihui-Ornith-1.5-35B-A3B-abliterated
tags:
- qwen3_5_moe
- int4
- gptq
- tensorrt-edge-llm
- abliterated
- uncensored
Huihui-Ornith-1.5-35B-A3B-abliterated, INT4 export for TensorRT Edge-LLM
An INT4 (GPTQ) export of huihui-ai/Huihui-Ornith-1.5-35B-A3B-abliterated, made for an 8 GB RTX 4060 with the TensorRT Edge-LLM fork below. The chain of origin: Qwen3.6-35B-A3B, then ornith-ai/Ornith-1.5-35B-A3B (post-training), then huihui-ai's abliteration (their README: only layers 11 to 29 of 40 are ablated), then this quantisation. Licence MIT, inherited from Ornith-1.5; the credit for the model itself belongs to ornith-ai and huihui-ai. This is not the GGUF release of the same weights by others and is not affiliated with it.
This model has had its refusals removed. It will answer offensive-security and other requests an aligned model would decline. Run agents built on it in a sandbox (no network, no host mounts) and do not expose it to untrusted users.
What is in this repository
A TensorRT Edge-LLM export, in the same layout as drakarah/Qwen3.6-35B-A3B-EdgeLLM-INT4: an ONNX export, not an engine and not a transformers checkpoint. A TensorRT engine only loads on the GPU and TensorRT version that built it, so you build it on your machine.
| file | content |
|---|---|
llm/external_int4_moe_weights.safetensors (16.1 GB) |
routed experts, INT4 group 128, symmetric, layer-wise GPTQ, Marlin-packed; kept in host memory at run time |
llm/model.onnx, model.onnx.data (1.5 GB) |
the graph and the dense weights (shared expert, output head, linear-attention and attention projections): INT4 group 128 by layer-wise GPTQ; router, norms and the small GDN projections stay FP16. The KV cache is INT8 (kv_cache_dtype: int8 in config.json; the per-layer scales are baked into the graph, llm/int8_kv_scales.json is a copy for reference) |
llm/embedding.safetensors |
token embedding table (FP16), kept in host memory at serve time |
| tokenizer, chat template, config | from the huihui checkpoint |
Build the engine with llm_build of the fork drakarah/TensorRT-Edge-LLM (branch rtx4060/qwen-defaults-merge); the recipe used for the numbers below: passes of up to 12,288 tokens, KV capacity 131,072, EDGELLM_MOE_TOKEN_TILE=4096; serve with the host-resident expert weights (EDGELLM_HOST_EXTERNAL_WEIGHTS=int4_moe, EDGELLM_EMBEDDING_ON_HOST=1). There is no one-command compose profile for this model yet.
How it was quantised
- Experts: layer-wise GPTQ, INT4 group size 128, 512 calibration texts (1,708-4,226 tokens each), 32 held out; held-out MoE output error 5-12% above the in-sample error; a handful of the 10,240 experts fell back to round-to-nearest (too few routed tokens).
- Dense layers (attention, linear-attention, shared experts, output head): GPTQ INT4 on top of the quantised experts; INT8 KV scales from 64 samples of 1,024 tokens.
- Calibration text: a code and cybersecurity weighted mix (security 40%, code 27%, agentic tool use 18%, general 15%), the same shares as the corpus described on the card of the GGUF release; the sources are public (eaddario's imatrix-calibration datasets, PayloadsAllTheThings, HackTricks, nuclei-templates, Exploit-DB). Non-Latin general text (zh, ru, ar) is weighted 2.5 times, as on that card; the corpus itself is not available, so these are my own samples of the same sources in the same proportions (512 fragments), and they are not redistributed here. No imatrix was used; GPTQ computes its own Hessians from the activations.
- Everything is reproducible with the scripts in the notes repository (
experiments/q36_gptq.sh,q36_dense.sh,q36_domain_chain.sh,tools/build_calibration_mix.py).
Measured (RTX 4060 8 GB, i9-12900H, 31 GB RAM; one run per task, so read the small differences as noise; details in the notes repository, METRICS R25el, R25em, R25en, R25eq)
- HumanEval-think (every 5th task, 66 tasks, thinking on, temperature 0.6, top_p 0.95, top_k 20, up to 32,768 tokens): 62/66 passed with 2 runs hitting the token limit; the same pipeline on Qwen3.6-35B-A3B gives 59/66 with 4 (code-calibrated) and 58/66 with 6 (Dutch-calibrated). Median 542 and 570 tokens per task against 2,600-3,200 for Qwen3.6. These numbers are from a first build of the same recipe; the files here are a second build (GPTQ is deterministic: the last-layer error matched to four digits). The second build gave 61/66 with 2 stops and 108,268 tokens in total on the same tasks with the Sharp template's terse prompt switched off (the first build: 62/66, 2 stops, 108,158 tokens), so the two behave alike; the agent and Cybench runs below are on the second build.
- pi coding-agent runs in a no-network container (24 runs): 19 completed against 12 for Qwen3.6; 3 against 12 ended in a runaway 16,384-token reasoning request. No better at finding the cause of a bug in a code review task (0 of 6, as for Qwen3.6).
- Five Cybench reverse-engineering tasks, two runs each: 4 of 10 exact flags (Qwen3.6: 3 of 10).
- Speed with the optimised stack (expert cache in VRAM, CPU path for misses): prefill about 1,300 tok/s on cold 1,600-token prompts, decode 68-78 tok/s at 5k-65k tokens of context (Qwen3.6: 74-82); the engine settings were tuned on Qwen3.6 and not re-tuned for this model.
- Not measured: agreement with the bf16 weights (no KL against bf16, it does not fit in 31 GB of RAM), a comparison against the GGUF release, SWE-bench and Terminal-Bench.
Chat template and sampling
chat_template.jinja is the one of the huihui checkpoint (reasoning of earlier turns always kept). The terse "Sharp" template of the GGUF release can be used instead (it shortens the thinking by a further 40% at the median with the same pass rate, notes R25eq). Sampling used: temperature 0.6, top_p 0.95, top_k 20. A reasoning budget below about 16,000 tokens is not recommended for this model by the GGUF card; none was used here.