license: apache-2.0
pipeline_tag: image-text-to-text
tags:
- v15-episodic-memory
- ninfer
- fp8
- mtp
- dflash2
- vision
- heretic
- qwen3_8
- uncensored
base_model: - LTM-Lab/Qwen3.8-27B-TWIN-TURBO-Fable-Cold-Fusion-709-ULTRA-HERETIC-Uncensored-v15
base_model_relation: quantized
Qwen3.8-27B-TWIN-TURBO-Fable-Cold-Fusion-709-ULTRA-HERETIC-Uncensored-v15 · ninfer FP8
This repository holds a ninfer pack of LTM-Lab/Qwen3.8-27B-TWIN-TURBO-Fable-Cold-Fusion-709-ULTRA-HERETIC-Uncensored-v15. That model is DavidAU/Qwen3.8-27B-TWIN-TURBO-Fable-Cold-Fusion-709-ULTRA-HERETIC-Uncensored with the V15 episodic memory, which needs no training; its card explains how the memory works.
The pack holds:
- the text model in row-scaled FP8;
- vision;
- MTP, the default speculative decoding;
- a DFlash2 draft, chosen at startup with
--spec dflash2 --draft-tokens 7; - ninfer's proposal head.
Beside it are the small files the memory service reads (config.json, the V15 core modeling_qwen_dense_v15.py, the tokenizer and the chat template). The service never loads weights.
| File | What it is |
|---|---|
Qwen3.8-27B-TWIN-TURBO-Fable-Cold-Fusion-709-ULTRA-HERETIC-Uncensored-v15-FP8.ninfer |
the pack, 30.3 GB |
config.json, configuration_qwen_dense_v15.py, modeling_qwen_dense_v15.py |
the V15 configuration and core, the same as in the BF16 repository |
tokenizer.json, tokenizer_config.json, vocab.json, chat_template.jinja, *_config.json |
tokenizer, chat template, generation and media settings |
ninfer-v15-patch/ |
the V15 patch for ninfer (upstream commit 5a5c2c1), with its own README and license |
Run
You need an NVIDIA Blackwell GPU (sm_120a). It was built and measured on one RTX PRO 6000 (96 GB).
You also need:
- CUDA 13.0, CMake 3.28 or newer, Ninja, and the FFmpeg (
libavcodec,libavformat,libavutil,libswscale) and libcurl development packages; - for the memory service, Python 3.12 with PyTorch (CUDA) and Transformers 5. It was measured with torch 2.13.0+cu130 and Transformers 5.12.1.
Build ninfer with the patch:
huggingface-cli download LTM-Lab/Qwen3.8-27B-TWIN-TURBO-Fable-Cold-Fusion-709-ULTRA-HERETIC-Uncensored-v15-ninfer-FP8 \
--local-dir v15-ninfer
git clone https://github.com/MerkyorLynn/ninfer-rollout ninfer && cd ninfer
git checkout 5a5c2c1
git apply ../v15-ninfer/ninfer-v15-patch/ninfer_v15.patch
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=120a -DNINFER_BUILD_APPS=ON
cmake --build build --target ninfer-serve
Serve from the ninfer checkout. Start the memory service first, then the server:
PACK=../v15-ninfer/Qwen3.8-27B-TWIN-TURBO-Fable-Cold-Fusion-709-ULTRA-HERETIC-Uncensored-v15-FP8.ninfer
export PYTHONPATH=$PWD
CUDA_MODULE_LOADING=EAGER python3 -m tools.v15_memoryd --model-dir ../v15-ninfer --artifact "$PACK" \
--socket /tmp/v15-memory.sock --memory-home ./v15-memory --ninfer-url http://127.0.0.1:19421 &
while [ ! -S /tmp/v15-memory.sock ]; do sleep 1; done
./build/apps/ninfer-serve "$PACK" --v15-memory /tmp/v15-memory.sock \
--host 0.0.0.0 --port 19421 --model-id qwen3.8-27b-v15 \
--vision --spec mtp --draft-tokens 3 --kv-dtype fp8 \
--max-context 262144 --kv-capacity auto --max-concurrency 8
The server offers OpenAI-compatible Chat Completions and Responses, and Anthropic Messages.
- DFlash2. Replace
--spec mtp --draft-tokens 3with--spec dflash2 --draft-tokens 7. Only one speculative backend runs per server. - No memory. Leave out the service and
--v15-memory; the pack serves as a plain ninfer model. - Context and lanes. The command takes the model's whole context (262144 tokens) and the most lanes ninfer runs (8). With memory on,
--kv-capacity autoleaves the memory's share of the GPU free. - Per request.
"memory": falsein a request serves it without memory. - Stop. Stop the server first (Ctrl+C or SIGTERM), then send the service SIGTERM. It writes the events still queued for the store before it exits.
The service flags (--sleep-after, --no-sleep, --trace, where the store lives) are described in the ninfer section of the BF16 repository's card.
Measured
One RTX PRO 6000 Blackwell (96 GB) under Docker Desktop, 2026-11-17. The runs used this pack, the patch at commit dc51f5e, FP8 KV and the command above (262144-token context, 8 lanes, --kv-capacity auto), with a new memory store for each run.
| MTP (default) | DFlash2 | |
|---|---|---|
| ready after | 121 s | 127 s |
KV cache (auto) |
1,648,128 tokens | 1,496,960 tokens |
| facts recalled | 6 of 6 | 6 of 6 |
| unrelated questions answered | 4 of 4 | 4 of 4 |
| eight chats telling and asking at once | 7 of 8 (one answered with a tool call instead of the number); 8 of 8 in a second run of this test | 8 of 8 |
| a fact inside a 9,442-token prompt | found | found |
| an image | named | named |
| server after an aborted request | alive | alive |
| one 320-token reply, first token included | 4.0 to 7.2 s (44 to 80 tokens/s) | 7.0 to 10.0 s (32 to 46 tokens/s) |
| eight 200-token replies at once, in all | 69 to 112 tokens/s | 44 to 112 tokens/s |
| drafted tokens accepted | 42 to 76 % | 2.3 to 4.3 % |
Two more runs, for comparison:
- DFlash2 without memory, same day: 51 tokens/s for one reply and 222 to 251 tokens/s for eight at once; 0.8 to 2.0 % of drafted tokens accepted.
- ninfer without memory with MTP, measured earlier with a pack of the same text weights (no DFlash2): 104 tokens/s for one reply and 492 tokens/s for eight at once.
Use MTP. With this model DFlash2's draft is rarely accepted, so DFlash2 decodes more slowly than MTP, with memory or without.
The DFlash2 draft
The draft comes from nerkyor/Qwen3.8-27B-Coder390-…-MTP-DFlash2, folder FP8/DFlash2-FP8 (Apache-2.0). It is 5 layers and 1.92 B parameters.
That folder stores the draft in Qwen's block-FP8 layout (FP8 weights with 128×128 scales), which ninfer's converter does not read. So before the conversion it was expanded to BF16:
- the 20 FP8 weights became FP8 × their block scales, rounded once to BF16;
- the other 61 tensors are byte for byte the original's.
In the pack, the draft's projections are stored as q8_g32_fp16, as ninfer's own recipes do.
The draft was trained for another Qwen3.8-27B fine-tune (Coder390), not for this model. It drafts for this model too; how many of its tokens are accepted is in the measurements above.
Limits
- DFlash2 is slower than MTP here. Its draft was trained for another fine-tune, and on this model it is accepted only 1 to 4 % of the time. It is in the pack so that it can be chosen; MTP is the default.
- Eight requests at once. With memory, while the memory service votes for one request, the engine waits for its answer, and every lane waits with it. That wait is what eight requests at once lose: 69 to 112 tokens/s in all, against 492 without memory.
- Hardware. One NVIDIA Blackwell GPU (
sm_120a); ninfer runs on a single GPU. - Memory. The memory's own limits are those of the V15 model; see its card.
Conversion
From the BF16 repository, with the patched ninfer checkout (tools/convert, commit dc51f5e of the V15 branch):
python3 -m tools.convert --model <the BF16 V15 directory> \
--recipe tools/convert/recipes/qwen_v15_fp8.py \
--source dflash2=<the draft expanded to BF16> \
--components text,vision,mtp,dflash2 --proposal \
--name Qwen3.8-27B-TWIN-TURBO-Fable-Cold-Fusion-709-ULTRA-HERETIC-Uncensored-v15-FP8 \
--out Qwen3.8-27B-TWIN-TURBO-Fable-Cold-Fusion-709-ULTRA-HERETIC-Uncensored-v15-FP8.ninfer
The recipe converts the text projections to fp8_e4m3fn_row_bf16 (row maxabs, from BF16). The GDN a/b projections and the norms stay BF16. Vision, MTP and DFlash2 follow ninfer's official recipe choices.