library_name: optimum-rbln
license: apache-2.0
pipeline_tag: image-text-to-text
base_model:
- Qwen/Qwen3.8-27B
- huihui-ai/Huihui-Qwen3.8-27B-abliterated
tags: - qwen3_5
- qwen3.8
- huihui
- abliterated
- multimodal
- rbln
- ca25
Huihui-Qwen3.8-27B-abliterated-RBLN-0.11.2
huihui-ai/Huihui-Qwen3.8-27B-abliterated, pre-compiled for 16 Rebellions RBLN-CA25
NPUs with optimum-rbln. This repository contains compiled artifacts, tokenizer
and processor configuration; RBLN NPUs and the RBLN SDK are required.
Full 64-layer prefill, batch-1 decode, and vision compilation completed successfully.
Runtime loading, generation, long-context accuracy, and device memory fit have
not been tested. This is a compile-validated release, not an inference benchmark.
Hardware and software
| Component | Configuration |
|---|---|
| NPU target | 16× RBLN-CA25; language model and vision each compiled across 16 devices |
| Compiler | rebel-compiler==0.11.2 |
| Wrapper | optimum-rbln==0.11.2 |
| Installed serving stack (not exercised) | vllm-rbln==0.11.2.post1, vllm==0.24.0+cpu |
| Framework | transformers==5.8.1, torch==2.11.0+cpu |
| Weights | Original BF16 checkpoint, loaded exactly into FP32 host tensors; no integer quantization added |
| Host graph dtype | float32; this is not a claim that all NPU arithmetic is FP32 |
Compile configuration
| Option | Value |
|---|---|
max_seq_len |
262,144 (source model's native configured maximum) |
batch_size |
1 |
decoder_batch_sizes |
[1] |
attn_impl / kvcache_partition_len |
flash_attn / 16,384 |
kvcache_num_blocks |
16 (262,144 shared KV token positions) |
prefill_chunk_size / gdn_chunk_size |
128 / 128 |
| Layers | 64: 48 GatedDeltaNet + 16 full-attention |
Vision max_seq_len / batch_size |
1,024 / 1 |
| Phases | prefill, decode |
| MTP | Excluded; no speculative decoding artifact |
Only a batch-1 decoder is included. There are no larger dynamic-batch decoder
variants in this release. Input and generated tokens share the context budget.
The compiled context length does not establish long-context quality or runtime fit.
Serving
The following is an untested starting configuration for the installed
vLLM-RBLN stack; runtime compatibility and memory usage still need validation.
export VLLM_PLUGINS=rbln
export VLLM_RBLN_USE_VLLM_MODEL=0
export VLLM_RBLN_NUM_DEVICES_PER_LOCAL_RANK=16
export VLLM_RBLN_SUB_BLOCK_CACHE=false
vllm serve iAcloud/Huihui-Qwen3.8-27B-abliterated-RBLN-0.11.2 \
--dtype float32 \
--max-model-len 262144 \
--max-num-seqs 1 \
--max-num-batched-tokens 128 \
--enable-chunked-prefill \
--no-enable-prefix-caching
Do not pass --tensor-parallel-size 16: the 16-device partition is baked into
the RBLN artifacts; the vLLM process uses its default tensor parallel size of 1.
Use a compatible runtime and retain the compiled prefill and batch dimensions.
Vision support is included in the compiled files but has not been exercised.
Provenance and validation
- Source: huihui-ai/Huihui-Qwen3.8-27B-abliterated
- Source revision:
739e3c5b89849f6c238ce1e5b70008612ae42cdd. - Loaded all 1,184 main-model parameter tensors; omitted the 15 auxiliary MTP tensors.
- Applied a process-local GDN prefix-sum patch: replace
cumsumwith the
equivalent triangular matrix multiplication to work around SDK TP lowering. - Preserved the original integer-free weight path; source BF16 values are exactly
representable in the FP32 host model. Compiler lowering controls device precision. - Full compilation completed in 990.9 seconds. Artifact payload is approximately
55.8 GiB; this file size is not a device-memory budget. - No NPU inference or runtime creation was performed.
compile_status.json records the local compilation result. Source model training,
post-training claims, and intended usage are documented by the original author.
License
Apache-2.0, as declared by the source repository. Credit for the model belongs
to huihui-ai and the Qwen contributors. This repository provides an RBLN compilation.