language:

• en
• zh
• de
  license: apache-2.0
  library_name: ninfer
  pipeline_tag: image-text-to-text
  base_model:
• huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF
• z-lab/Qwen3.8-27B-DFlash2
• Qwen/Qwen3.8-27B
  base_model_relation: merge
  tags:
• qwen3.8
• qwen3.8-27b
• ninfer
• ninfer-v3
• abliterated
• swift-1.5
• gguf-blocks
• gsq-rco
• iq3_s
• dflash2
• mtp
• speculative-decoding
• multimodal
• vision
• uncensored

Huihui Qwen3.8-27B — Abliterated Swift-1.5 GSQ-RCO IQ3_S + DFlash2 (NInfer v3)

Community conversion and component combination, not a newly trained model. The NInfer v3 artifact combines the Huihui abliterated Swift-1.5 GSQ-RCO IQ3_S GGUF text weights (including MTP), a pretrained Qwen3.8-27B DFlash2 drafter, and the Qwen3.8-27B vision projector.

RTX 4090 local benchmark (19 October 2026): up to 229.04 output tokens/s in a 2,048-token single-request decode test (32K input), 191.78 tokens/s at 160K input, and 63/63 successful HTTP requests. These are client-side measurements of the entire server, not isolated DFlash2 speedups. See BENCHMARK.md for methodology and limits.

Artifact details

This file is not a standard Transformers checkpoint and cannot be loaded directly with transformers.AutoModel.from_pretrained.

Quick start (Windows / PowerShell)

Download (after the model file has been uploaded to this repository):

hf download SPK1337/Huihui-Qwen3.8-27B-Swift1.5-Abliterated-IQ3S-DFlash2-NInfer `
  Huihui-Swift15-Abliterated-IQ3S-DFlash2.ninfer `
  --local-dir C:\ninfer-all\models

Start NInfer-all:

cd C:\ninfer-all

.\build-ninja\apps\ninfer-serve.exe `
  ".\models\Huihui-Swift15-Abliterated-IQ3S-DFlash2.ninfer" `
  --host 127.0.0.1 `
  --port 8080 `
  --model-id qwen3.8-27b `
  --max-context 32768 `
  --kv-capacity auto `
  --max-concurrency 1 `
  --kv-dtype rk4v4 `
  --gdn-state-fp16 `
  --spec dflash2 `
  --draft-tokens 5

For vision support, add --vision --vision-residency overlay if your runtime and GPU memory allow it. For an MTP run, replace --spec dflash2 --draft-tokens 5 with --spec mtp --draft-tokens 3. Restart the server after changing speculative backends.

Example JSON API request. Explicit UTF-8 bytes avoid encoding issues in Windows PowerShell 5.1:

$body = @{
  model = 'qwen3.8-27b'
  messages = @(@{role='user'; content='Explain speculative decoding in two sentences.'})
  enable_thinking = $false
  max_tokens = 512
} | ConvertTo-Json -Depth 10 -Compress

$bytes = [System.Text.Encoding]::UTF8.GetBytes($body)
$r = Invoke-RestMethod -Uri 'http://127.0.0.1:8080/v1/chat/completions' `
  -Method POST -ContentType 'application/json; charset=utf-8' -Body $bytes
$r.choices[0].message.content

RTX 4090 benchmark — 19 October 2026

The publisher supplied the raw results ZIP for a local full-suite run, label HUIHUI-SWIFT15-IQ3S-DFLASH2-16GB, run ID 20261019-205035-980161.

• Hardware: NVIDIA GeForce RTX 4090 (24 GiB class), Windows, NInfer.
• Suite: 63/63 HTTP requests succeeded; two narrow 100K/160K needle-retrieval probes passed.
• Configuration reported for the run: 176,128 maximum context; rk4v4 KV; maximum concurrency 2; thinking disabled; 2,048 tokens per sustained decode test.
• Caveat: No server request JSONL was included in the ZIP. Draft acceptance counts, actual DFlash2 speedup versus MTP/no-speculation, and per-request server prefill statistics are not established by this benchmark.

The highest number (229.04 tok/s) is one observed run, not a multi-run average, peak hardware throughput guarantee, or a direct DFlash2-versus-MTP comparison. Prompt processing increases substantially at long contexts. Exact prompt repetition showed substantially lower time to first token, consistent with reuse, but cache metrics were not captured.

Full report: BENCHMARK.md. To obtain acceptance rates for a later run, add --request-log-jsonl PATH when launching ninfer-serve, and inspect request_done.speculative or /metrics counters.

Provenance and reproduction

All model components and training credit belong to their original authors:

1. Text and MTP GGUF: huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF — Huihui-Qwen3.8-27B-abliterated-Swift-1.5-GSQ-RCO-IQ3_S-mtp.gguf.
2. DFlash2 drafter: z-lab/Qwen3.8-27B-DFlash2, adapted from the original Qwen3.8-27B distribution.
3. Vision projector: ggml-org/Qwen3.8-27B-GGUF, mmproj-Qwen3.8-27B-BF16.gguf.
4. Architecture/config/tokenizer: Qwen/Qwen3.8-27B.
5. Converter/runtime: iamwavecut/ninfer-all, using the qwen3_8_27b_gguf recipe.

Key conversion options:

--recipe qwen3_8_27b_gguf
--model <Qwen3.8-27B tokenizer-and-config-directory>
--source gguf=<Huihui Swift-1.5 GSQ-RCO IQ3_S MTP GGUF>
--source dflash2=<z-lab DFlash2 model directory>
--source vision=<Qwen3.8 mmproj BF16 GGUF>
--components text,vision,mtp,dflash2
--resource chat_template.jinja=tools/chat_templates/qwen3_8.jinja
--proposal
--device cpu
--name huihui-qwen3.8-27b-swift15-iq3s

The local conversion produced a single 13.99-GiB file, 1,249 objects, in 143.7 seconds (conversion time only). Existing quantized text weights were imported as GGUF blocks; this release does not claim new fine-tuning or drafter training.

Limitations and responsible use

• The DFlash2 drafter was trained for the original Qwen3.8-27B rather than specifically on the abliterated Swift-1.5 weights; acceptance/quality may differ.
• A functioning text API and successful client benchmark do not prove that DFlash2 improves speed relative to MTP or plain decoding; a controlled A/B test is still needed.
• Quantized and abliterated weights can change response behavior or accuracy compared with upstream models. No guarantee of reliability or suitability for safety-critical tasks is provided.
• Only two narrow long-context retrieval checks are represented here; they are not a broad evaluation of reasoning, safety or instruction following.
• Vision capability is packaged but was not independently evaluated in this benchmark.

License

Apache License 2.0. See the repository's LICENSE and NOTICE for author and licensing attribution, and review the respective upstream model cards. This repository is a derivative community conversion, not an official Qwen, Huihui, z-lab, or NInfer release.