language:

• en
• de
• zh
  license: apache-2.0
  library_name: ninfer
  pipeline_tag: image-text-to-text
  tags:
• qwen3.8
• qwen3.8-27b
• huihui
• uncensored
• abliterated
• ninfer
• ninfer-v3
• gguf-blocks
• gsq-rco
• iq3_s
• mixed-precision
• vision
• multimodal
• dflash2
• mtp
• speculative-decoding
• cuda
• rtx-4090
• sm89

Huihui Qwen3.8 27B — Uncensored · GSQ-RCO IQ3_S · Vision · DFlash2 · MTP · NInfer

An RTX 4090 (SM89)-tuned NInfer release of the Huihui Qwen3.8 27B uncensored/abliterated variant. The community build and launch profile target NVIDIA GeForce RTX 4090 (24 GB) with CUDA 13.4 and provide Vision, DFlash2 and Multi-Token Prediction (MTP) support. This is the non-Swift1.5 edition.

Real-world RTX 4090 measurements: Across 1,831 completed requests in two recorded sessions, the median per-request generation rate was 146.6 tokens/s (90th percentile 215.5 tokens/s). See the benchmark and methodology. These are observed results, not a controlled comparison or a guarantee of performance.

[!IMPORTANT]
This repository distributes a .ninfer model artifact, not a standard Transformers/Safetensors or directly loadable GGUF checkpoint. You need a compatible NInfer-all runtime (or the SPK1337 / spk-c fork). DFlash2 and MTP are alternative speculative decoding modes, not two modes that must be enabled simultaneously.

At a glance

Quantization note: IQ3_S identifies a GGUF weight format; it does not mean that every tensor, the Vision encoder, or the speculative decoding components use an identical three-bit format. GPU memory needs depend on the embedded components, KV-cache format, context length and active speculation mode.

Files

The files above are the intended release layout; optional files may appear only after they have been uploaded.

Quick start — Windows PowerShell

1. Install the download CLI

py -m pip install --upgrade huggingface_hub

Download the model into a local models directory:

$repo = 'SPK1337/Huihui-Qwen3.8-27B-Uncensored-GSQ-RCO-IQ3S-Vision-DFlash2-MTP-NInfer'
$file = 'Huihui-Qwen3.8-27B-GSQ-RCO-IQ3S-Vision-DFlash2.ninfer'
hf download $repo $file --local-dir '.\models'

2. Start the model with DFlash2

You need a compatible Windows build of ninfer-serve.exe. Adjust $server to match your executable location; the example below uses the build directory from the SPK RTX 4090 setup.

$server = 'C:\coding\ninfer-all\build-rtx4090-coding\apps\ninfer-serve.exe'
$model  = (Resolve-Path '.\models\Huihui-Qwen3.8-27B-GSQ-RCO-IQ3S-Vision-DFlash2.ninfer').Path

& $server $model `
  --model-id huihui-qwen3.8-27b `
  --host 127.0.0.1 --port 8080 `
  --max-context 32768 --kv-capacity auto `
  --kv-dtype rk4v4 --gdn-state-fp16 `
  --spec dflash2 --draft-tokens 5

Vision (image inputs): add --vision to the command if the artifact contains a valid Vision component. For tight GPU-memory budgets, consider --vision-residency overlay and a smaller context limit.

3. Alternatively, use MTP

Replace the speculation flags above with:

--spec mtp --draft-tokens 3

MTP and DFlash2 are different selectable backends. The published RTX 4090 workload measurements below used DFlash2 with five draft tokens, not MTP. Actual performance depends on the runtime build, hardware, model components, context length, prompts and draft count.

4. Send a request

The server exposes an OpenAI-compatible API at http://127.0.0.1:8080/v1.

$payload = @{
    model = 'huihui-qwen3.8-27b'
    messages = @(@{role = 'user'; content = 'Hi! Introduce yourself in one sentence.'})
    max_tokens = 128
} | ConvertTo-Json -Depth 10

Invoke-RestMethod `
    -Uri 'http://127.0.0.1:8080/v1/chat/completions' `
    -Method Post `
    -ContentType 'application/json' `
    -Body $payload

To process images, start the server with --vision and use the NInfer-supported multimodal request format. Vision is disabled by default in the engine.

RTX 4090 optimization and benchmark

This package was configured, calibrated and measured for the NVIDIA GeForce RTX 4090 (Ada Lovelace, compute capability 8.9 / SM89) using a custom Windows NInfer build and an RTX 4090 device profile (rtx4090-cuda134-sm89.json). The model weights themselves are not exclusive to the RTX 4090; the optimization refers to the runtime/build, profile and deployment settings. Other supported NVIDIA GPUs may work with their own profiles and memory budgets.

Measured inference — RTX 4090, CUDA 13.4

Data and method: The figures come from an RTX 4090 NInfer server JSONL request/throughput log recorded on October 30–31, 2026 (UTC), over two server starts. We excluded 10 cancelled requests and calculated per-request generation as completion_tokens / timings_seconds.decode for completed requests with positive decode time. Prefill uses computed_prefill_tokens / timings_seconds.prefill, which includes actual computed tokens rather than already cached prompt tokens. Percentiles are calculated over individual requests. The weighted decode figure is total generated tokens divided by the sum of their recorded decode times; it is not end-to-end user-facing throughput.

The observed configuration was RTX 4090 / CUDA 13.4 / SM89, .ninfer artifact size 13.99 GiB, --max-context 176128, --kv-capacity auto, --kv-dtype rk4v4, --gdn-state-fp16, --max-concurrency 2, --spec dflash2 --draft-tokens 5, --prefill-chunk 1024, host cache 16 GiB and --vision --vision-residency overlay. No requests contained media, so these numbers do not measure Vision latency or accuracy. MTP was not benchmarked in this log.

[!NOTE]
This is a real-world workload observation, not a standardized isolated synthetic test or an A/B comparison to unmodified NInfer, other quantizations or other GPUs. The log shows this setup running on an RTX 4090; it does not establish a numerical acceleration due solely to the custom build/profile. Request concurrency, caching, prompt lengths, sampling and draft acceptance vary. Results are not guaranteed on other configurations.

For the exact capture setup, aggregation rules, details and a benchmark-oriented launch example, see BENCHMARK-RTX4090.md.

Hardware compatibility

• Primary validated target: RTX 4090, 24 GB, SM89, CUDA 13.4-compatible driver/build and matching device profile. You can use the conservative 32,768-token launch example above as a starting point.
• The measured runs configured up to 176,128 context tokens. This is a configured context ceiling, not a promise that any workload at that length will fit in memory with every optional feature enabled.
• Other GPUs require suitable NInfer binaries/profiles and sufficiently large GPU/host-memory budgets. See supported GPUs, launch recipes and the SPK fork.

Model lineage and conversion provenance

This release is presented as a Huihui Qwen3.8-27B uncensored, GSQ-RCO IQ3_S, Vision + DFlash2 + MTP NInfer package, without Swift1.5. The relevant upstream projects are:

1. Base model: Qwen/Qwen3.8-27B — original Qwen architecture and weights.
2. Uncensored / abliterated variant: huihui-ai/Huihui-Qwen3.8-27B-abliterated — the Huihui abliterated release.
3. GSQ-RCO quantization reference: ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF — GSQ-RCO GGUF IQ3_S family and related quantization methods.
4. DFlash2: z-lab/Qwen3.8-27B-DFlash2 — speculative draft component.
5. Inference and conversion tooling: NInfer-all and spk-c/ninfer-all.

Provenance verification: These are relevant upstream projects, not a claim that every named checkpoint was necessarily combined verbatim. A model filename cannot establish whether IQ3_S text weights were quantized from Huihui or from ordinary Qwen, nor whether all auxiliary heads were preserved. Inspect the adjacent *.conversion.json report, converter inputs and—where available—tensor provenance before asserting the precise merge or the degree of abliteration. Quantizing an unmodified Qwen checkpoint does not itself create the Huihui/uncensored variant. This distinction matters for correct attribution and user expectations.

NInfer's documented GGUF conversion recipe demonstrates how GSQ-RCO text weights, Vision, MTP and DFlash2 can be placed into one NInfer artifact. It is an illustrative reference, not a verified record of the command used for this particular file.

Integrity checks

If a SHA256SUMS file has been published, compare its entry with the downloaded model:

Get-FileHash '.\models\Huihui-Qwen3.8-27B-GSQ-RCO-IQ3S-Vision-DFlash2.ninfer' -Algorithm SHA256

Recalculate checksums whenever any artifact is modified or replaced. The conversion JSON is supplementary metadata and is not required at runtime.

Limitations and responsible use

• Abliterated / uncensored does not mean unrestricted capability or guaranteed compliance. It is a model variant description, not a safety, quality or performance guarantee.
• The model can produce inaccurate, biased or unsafe output. Verify factual claims, especially in high-stakes settings.
• The model has not been independently evaluated here for safety, factuality, coding quality, multilingual performance, Vision accuracy or benchmark scores.
• Follow applicable laws, privacy requirements and original upstream licenses. Do not upload private image/data content to untrusted endpoints.
• This community release is not an official product of Qwen, Huihui, ISTA-DASLab, z-lab or the NInfer maintainers; mentioning them is attribution, not endorsement.

Credits and license

Packaging / publication: SPK1337. All credit for upstream models, quantization approaches, speculative decoders and runtime development belongs to their respective authors and contributors. Please consult NOTICE, LICENSE and the upstream model cards for the applicable terms and acknowledgments.

This repository uses the Apache-2.0 license designation in reliance on the upstream license metadata identified above; actual redistribution must preserve all applicable attribution and license requirements. The NOTICE document provides attribution and does not replace any required license texts.

Reference documentation: NInfer-all README · Launch recipes · GGUF conversion · SPK fork