base_model: huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated
base_model_relation: quantized
license: apache-2.0
pipeline_tag: image-text-to-text
tags: [orinfer, jetson, e8p, int8, mixed-precision, safetensors, abliterated, uncensored]
Huihui Qwen3.6-35B-A3B abliterated · Orinfer E8P Q4A8
Quantized from huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated for Jetson AGX Orin 64GB, CUDA SM87, using Orinfer v0.2.1. The source is huihui-ai's uncensored, abliterated Qwen3.6-35B-A3B model. Its Qwen3.5 MoE architecture has 40 layers, 256 routed experts and 8 active experts per token. Text, images, multiple images, thinking, MTP, prefix caching and continuous batching use the native Orinfer runtime.
Two-stage integer E8P Q4, coefficients [4,1], signed-block128 rotation, FP16 output-row scales, beam8 encoding and INT8 computation. Embedding, output head, target experts and eligible vision matrices use E8P Q4. Sensitive attention output and MTP projections retain W8; norms and small sensitive tensors retain floating precision. Quantization reads original BF16 weights directly, with seed 20261002 and no activation calibration.
Read-only weight payload: 18,914,729,024 bytes, including scales, constants and precision exceptions. The aligned weight arena adds 12,992 bytes, totaling 18,914,742,016 bytes, 4.209 bits per recorded weight parameter. Weight parameters: 35,951,822,704. Context capacity: 262,144 tokens; KV is group-64 INT8 and persistent GDN state is FP32. This custom quantization and its physical K128 layout run with Orinfer.
Deploy
Download this repository into ./model:
hf download immiq/Huihui-Qwen3.6-35B-A3B-abliterated-Orinfer-E8P-Q4A8 --local-dir ./model
curl -fLO https://github.com/iMMIQ/orinfer/releases/download/v0.2.1/orinfer-v0.2.1-linux-aarch64.tar.xz
curl -fLO https://github.com/iMMIQ/orinfer/releases/download/v0.2.1/orinfer-exec-qwen35-35b-a3b-a8-sm87-v0.2.1.tar.xz
curl -fLO https://github.com/iMMIQ/orinfer/releases/download/v0.2.1/SHA256SUMS
sha256sum --ignore-missing -c SHA256SUMS
tar -xf orinfer-v0.2.1-linux-aarch64.tar.xz
./orinfer-v0.2.1-linux-aarch64/bin/orinfer package install ./orinfer-exec-qwen35-35b-a3b-a8-sm87-v0.2.1.tar.xz
./orinfer-v0.2.1-linux-aarch64/bin/orinfer validate-model ./model
./orinfer-v0.2.1-linux-aarch64/bin/orinfer serve ./model --model huihui-35b --cuda-graph decode_only --mtp-drafts 3
Execution-package digest: 25a611471edd681f2867f198c90be6a0c8e41b28d3ca1eab3d5317660e7e98c1. The configuration, buffer layouts and arithmetic signature match the released 35B package; every model tensor and frontend asset is bound to this source checkpoint. The original configuration is normalized by making the default empty mlp_only_layers explicit and retaining partial_rotary_factor=0.25 in rope_parameters.
The OpenAI-compatible Chat Completions endpoint is http://localhost:8088/v1/chat/completions, model ID huihui-35b. Set enable_thinking as needed and pass images through image_url. See the serving guide.
Validation
Validated on Jetson AGX Orin 64GB with the actual quantized weights: full model/tensor/kernel hashes, lossless K128 code roundtrips, request isolation, continuous batching, committed KV/GDN/convolution state, prefix restore, cancellation/arena reuse, MTP generation and shared target verification, graph replay with changing inputs, and streaming API requests with single/multiple images.
Original BF16 teacher-forced reference, fixed seed 20261002, 166 scored target positions across 12 text tasks and a 2048-token retrieval input: mean target NLL delta +0.00898 nats/token. Token agreement is a diagnostic. Free-generation API checks cover instruction following, arithmetic, concurrency and simple image colors. API readiness with full weight verification: 23.53 seconds in this run. One complete natural text request achieved 65.83 tokens/s including prefill and client wall time. The same fixed histories also pass original-BF16 target-NLL acceptance at batch 16 for ordinary decode and joint prefill. The separate serial-versus-cold-mixed quantized diagnostic does not pass its KL gate: mean KL 0.11933, threshold 0.05, selected-token NLL delta +0.00982, and identical text output in that case. A single-image case also differed in exact output. This was reproduced with v0.2.1; the test threshold was preserved. The diagnostic is retained in validation.json. Local model files were staged on eMMC; startup depends on storage and file-cache state. These are bounded acceptance checks; full-context quality and broad task benchmarks remain unmeasured for this checkpoint. See validation.json for the compact measurements and quality-report.json for case-level results.
The separate natural-generation set passed 10/11 automatically scored tasks, including 2K retrieval, JSON, instruction correction and simple image colors. It answered 683 for 37×24−19×13 (correct: 641). Python and Rust functions passed 5 and 4 functional tests respectively; the Python answer included a Markdown code fence. Translation and summary checks retained the requested meaning. These observations are included in validation.json.
License
Model weights: Apache-2.0. Source model credits: huihui-ai and Qwen. Original model card. Orinfer and its separate execution package use LGPL-3.0-or-later; source and third-party notices accompany the GitHub release.