← back to catalog · registered 2026-11-16 12:03

LTM-Lab/Qwen3.8-Flash-Next-FP4-Local-Abliterated-v15

LTM-Lab multimodal second-order
Your rig guess connected
? Why do I need an app?
Reading your rig…

This is a rough estimate. Install the free app - we'll show exact numbers.

Reading real hardware from your app right now. Numbers below are exact.

Below is the per-quantization compatibility for this model.

curl -H "Authorization: Bearer $ABL_KEY" \
     "https://abliteration.org/api/v1/models/LTM-Lab%2FQwen3.8-Flash-Next-FP4-Local-Abliterated-v15"
Response includes
  • classification m1
  • files 23
  • author_summary 17 models
  • readme_text full
10 credits · hourly refresh · ~4 KB payload Get an API key →
Abliteration classifier · v1.0.0
M1
Primary method

Direct removal

No other method signals detected in this model.
Confidence
MEDIUM
Why this label 3 signals
Method inferred from partial signals - repository name, related files, or tag patterns. Producer identity not confirmed; label may sharpen or shift as we gather more evidence.
  • 'abliterated' in name/tags
  • is_gguf=0 (base model)
  • no specific method indicators - defaulting to M1 (most common)
Refusal direction extracted via
Extraction technique

Difference-of-means

Confidence
MEDIUM
Why we say so
primary_method=M1; difference-of-means is the reference extraction for M1/M3 (Arditi 2024)
Downloads · 30-day
86
Likes
0
Descendants
2
in 1 direct fork
Model age
5d ago
created 2026-11-16

Training datasets

2 of 2 in /datasets

Corpora the author lists in the model card. Datasets tracked in our /datasets catalog carry a category badge linking to the workflow stage. Others open on Hugging Face.

Downloads over time
Now 86 → from 0 ↑ 0%
0326395 0 on Nov 16 86 on Nov 22 Nov
Nov 16 → Nov 22 · 7 snapshots · spans 6 days

Genealogy 1 direct fork

Full fork graph →

This model's place in the market. Above: what it was derived from. Below: the tree of everything derived from it.

Metadata

License
other
Tags
safetensors qwen4_exp v15-episodic-memory qwen3.8 qwen3.8-flash-next abliterated heretic nvfp4 mxfp8 modelopt mtp dgx-spark

Related

Total size
0 B
Files
23
Quantizations
1
Registered
2026-11-16 12:03
Last updated on HF
2026-11-17 03:14

Files by quantization

Auxiliary files 23 files 76.1 MB
model.safetensors.index.json 44.4 MB 96e5ea03 download
tokenizer.json 12.2 MB 0997f410 download
frozen-directions.json 7.15 MB 71116366 download
vocab.json 6.41 MB 0aa0ce06 download
merges.txt 3.20 MB a494e019 download
frozen-row-norms.json 2.31 MB 7995d02c download
modeling_qwen4_exp_v15.py 264 KB aba2f0e7 download
config.json 75.1 KB 8f06e8ae download
README.md 37.7 KB c63e3b1d download
tokenizer_config.json 17.5 KB 5de744b3 download
configuration_qwen4_exp_v15.py 10.5 KB 0ebfa2cb download
chat_template.jinja 8.74 KB c0c686f9 download
SHA256SUMS 8.12 KB d65bc216 download
LICENSE 7.87 KB 1d59e3ac download
release-files.json 6.76 KB aa390a13 download
provenance.json 4.47 KB 18d56020 download
serve-spark.sh 3.39 KB 5d093d1a download
NOTICE 2.57 KB 8d0451d0 download
bake-selection.json 556 B 9f08f390 download
preprocessor_config.json 390 B 2ea84a43 download
video_preprocessor_config.json 385 B 3ba673a5 download
generation_config.json 202 B 023756cf download
.gitattributes 166 B f473a276 download

README current version from Hugging Face


license: other
license_name: nvidia-open-model-agreement
license_link: LICENSE
pipeline_tag: image-text-to-text
datasets:

  • mlabonne/harmful_behaviors
  • mlabonne/harmless_alpaca
    tags:
  • v15-episodic-memory
  • qwen3.8
  • qwen3.8-flash-next
  • abliterated
  • heretic
  • nvfp4
  • mxfp8
  • modelopt
  • mtp
  • dgx-spark
  • vllm
    base_model:
  • DevelopingDad/Qwen3.8-Flash-Next-FP4-Local-Abliterated

Qwen3.8-Flash-Next-FP4-Local-Abliterated-v15

繁體中文摘要:這是 DevelopingDad/Qwen3.8-Flash-Next-FP4-Local-Abliterated 的 V15 轉換版。權重與原模型逐位元相同(FP4-Local 格式不變),多的是記憶的設定、兩個程式檔,以及一份鏡像補丁 vllm-v15-patch/:在 local-inference-lab 的 vLLM 鏡像(ghcr.io/local-inference-lab/vllm:release-10152026,能跑 FP4-Local 格式的 b12x 版 vLLM)上加入記憶;建好的鏡像是 lovewin10/vllm-v15:release-10152026-v15。用這個鏡像服務時,模型帶有跨請求保存的情節記憶:在一段對話裡說過的事,之後在另一段全新的對話裡問得到。V15 與 V14 相比拿掉了世界模型,多了睡眠:伺服器閒置時,用被服務的模型自己整理記憶庫(重播、合併重複、判斷新事實是否取代舊事實、建立相關事件之間的橋),不訓練、不寫入新內容。CUDA graph、前綴快取、MTP 推測解碼與非同步排程照常開著。實測結果與已知的限制寫在下面。

DevelopingDad/Qwen3.8-Flash-Next-FP4-Local-Abliterated converted to the V15 episodic memory architecture (Qwen4ExpV15ForConditionalGeneration).

  • The weights are the original's, byte for byte, in its FP4-Local format (ModelOpt mixed precision, NVFP4 experts with compressed scales). Nothing was trained, merged or re-quantized here; the memory adds no parameters.
  • Added: the architecture name, an auto_map and the memory options (qwen4_exp_v15_memory_config) in config.json, two code files (configuration_qwen4_exp_v15.py, modeling_qwen4_exp_v15.py) that hold the memory, and the image patch vllm-v15-patch/. SHA256SUMS lists this repository's files; the weights' lines are the original's.
  • Served by the image lovewin10/vllm-v15:release-10152026-v15 (built from vllm-v15-patch/), the model has a memory that persists across requests: what was said in one chat is answered in a later, separate chat. The store is on disk and survives restarts.
  • The image is ghcr.io/local-inference-lab/vllm:release-10152026 (the vLLM that serves this checkpoint's format with b12x kernels) with the memory added; it is not an official vLLM or local-inference-lab image.

How the memory works

While a prompt is read, the keys and values of its 12 indexed-attention (QSA) layers are cut into events of up to 128 tokens and written to a store on disk, each with a low-rank engram of the state it leaves in the 36 Gated DeltaNet layers. A later prompt is read once to see what it asks for; the stored events whose keys would take its attention are recalled by one vote for all layers and heads (with one hop along the bridges a sleep confirmed, and an event a sleep found superseded replaced by the one that supersedes it); their engrams are composed into the request's recurrent state and the prompt is read again with the recalled keys and values merged into the attention (by the local attention's log-sum-exp, which the patched b12x QSA kernels return). Generated tokens attend the recalled events too, inside the decode CUDA graphs. Nothing is retrieved as text and nothing is added to the prompt.

While no user request has come for 10 minutes (VLLM_V15_SLEEP_AFTER), the server sleeps: a thread of the engine process replays the events no sleep has seen yet through the server's own API with the memory read-only, merges repeated ones, takes out of recall a note the model says twice only asks something, asks the model in two wordings whether a later note changes an earlier one (two yeses retire the earlier one: it is then recalled only with the later one, before it), dreams a question each event answers and asks which person, thing or place two events it calls up together both name (a name written in both is a bridge), and downscales the strength of what was not used. It changes only how the store is organised: no training, no new content, no event deleted. A user's request stops a cycle at its next step. Every cycle writes one V15 sleep: line to the log.

Serve

Build the image from this repository's folder (the build pulls ghcr.io/local-inference-lab/vllm:release-10152026, about 44 GB, and stops if any file the patch replaces differs from the original it was made from), or pull it when it is on Docker Hub:

docker build -t lovewin10/vllm-v15:release-10152026-v15 vllm-v15-patch
# or: docker pull lovewin10/vllm-v15:release-10152026-v15

It starts as NVIDIA's command for one RTX PRO 6000 Blackwell starts its base (PROFILE=qwen38-flash-next); only the image and the model folder change, and a folder for the store is mounted at /memory. On one 96 GB GPU (under Docker Desktop, where the port is published instead of --network host):

docker run -d --name flash-next-v15 --init --gpus device=0 `
  --ipc host --shm-size 32g --stop-timeout 30 `
  --ulimit memlock=-1 --ulimit stack=67108864:67108864 -p 127.0.0.1:8000:8000 `
  -e PROFILE=qwen38-flash-next -e MODEL=/models/qwen -e PORT=8000 `
  -v <this repository>:/models/qwen:ro -v flash-next-cache:/cache -v <a folder for the store>:/memory `
  lovewin10/vllm-v15:release-10152026-v15

The profile carries the rest of NVIDIA's settings: a 262,144-token context, 16 sequences, FP8 KV cache, MTP with 3 draft tokens, b12x kernels, GPU memory utilization 0.96, the qwen3 reasoning parser and the qwen3_xml tool parser. The model is served as Qwen3.8-Flash-Next, with the sampling defaults of generation_config.json (temperature 1.0, top-p 0.95, top-k 20); thinking is on unless a request sets chat_template_kwargs.enable_thinking to false.

  • VLLM_V15_MEMORY=0 serves the folder as the plain checkpoint (no memory), exactly as the base image does.
  • VLLM_V15_SLEEP=0 keeps the memory and turns its sleep off; VLLM_V15_SLEEP_AFTER (default 600) is the idle time in seconds before a cycle starts. The sleep calls the server at http://127.0.0.1:$PORT; set VLLM_V15_SELF_URL when it listens elsewhere, and give an API key as VLLM_API_KEY (not --api-key), which the sleep sends as well.
  • VLLM_V15_MAX_SLOTS (default 4): requests that read the memory at once; each holds about 100 MiB of decode buffers taken from the KV cache. Other requests of the same step run without memory, the ones with a slot keep theirs.
  • VLLM_V15_INDEX_GB (unset by default): the store's index rows are reserved at start-up for the payloads it holds and 8,192 more (about 0.53 GiB for a new store, at most 2 GiB), and the log warns when they are 80% full (a restart sizes them anew); a number pins the size. Beyond the rows a vote reads the rest back from the store.
  • VLLM_V15_WORK_GB (default 1.25): GPU memory held back from the KV cache for the memory's work the vLLM profile does not see (a prefill's recurrent states, the keys of recent votes, the far contexts of the requests that recall).
  • VLLM_V15_KV_CACHE_GB (default 0.25): GPU memory for the decoded keys of recently recalled events.
  • VLLM_V15_TRACE=1 logs every recall decision.
  • With the memory on the KV cache is smaller (669,427 tokens instead of 766,241 with the defaults above, measured). The GPU still shows nearly full, as with the base image: the profile hands everything below --gpu-memory-utilization 0.96 to the KV cache. A lower -e GPU_MEMORY_UTILIZATION (or a fixed -e KV_CACHE_MEMORY_BYTES=..., the base image's options) leaves GPU memory free, all of it taken from the KV cache: about 0.96 GiB, some 52,000 tokens, for every 0.01. Below about 0.90 one 262,144-token request no longer fits and vLLM stops after loading the weights, so add -e MAX_MODEL_LEN=-1 as well (vLLM serves the longest context that fits and logs it as Auto-fit max_model_len) or a smaller MAX_MODEL_LEN. These figures are worked out from the V14 image's start-up log at 0.96, not measured at the lower settings.
  • Other images do not know the V15 architecture name, including the DGX Spark image of the original's serve-spark.sh: there, serve the original repository (or set architectures in config.json back to Qwen4ExpForConditionalGeneration). vllm-v15-patch/README.md lists what the patch changes.

Measured

One RTX PRO 6000 Blackwell (96 GB) under Docker Desktop, 2026-11-16, with the published image, the profile above, an empty store and -e VLLM_V15_SLEEP_AFTER=20 (so that the check sees a sleep); every statement and every question a chat of its own, temperature 0, thinking off. A handful of questions per check: a functional check, not a benchmark.

Memory on (V15) Memory off (the base image, measured with the V14 image's memory off)
A fact told in one chat, asked in a new chat (6 facts) 6 / 6 0 / 6
Unrelated questions asked afterwards (4) 4 / 4 4 / 4
The six facts asked again after one sleep 6 / 6
Decode speed, three answers of 266 to 400 tokens 119.8 tokens/s 122.4 tokens/s
First token of those three short prompts 1.03 s 0.15 s
KV cache 669,427 tokens 766,241 tokens
Ready after (weights on an SSD) 1779 s

The sleep cycle took 87.8 s (49 events replayed, 3 told twice merged, no correction among these facts and none made, 3 bridges); no traceback was logged. The memory's index takes 0.53 GiB of the device at start-up. Against the V14 image on this checkpoint (99.0 tokens/s, first token 1.06 s, KV cache 656,518 tokens), V15 decodes faster: the event V14's world model expected next is no longer attended.

The V15 memory core was also measured on Qwen3.5-0.8B with Hugging Face Transformers, one sleep between telling and asking against no sleep: single facts 9/10 both ways, facts among 48 other notes 7/10 both ways, among 248 6/10 both ways, corrected facts 1/6 without and 2/6 with the sleep, two-step questions 5/6 without and 6/6 with it, long prompts 3/3, unrelated questions 7/8. Corrections and two-step questions were measured on the 27B FP8 conversion with the SGLang image (its card): after a sleep 6 of 6 corrected facts answered with the new value, and every pair a two-step question needs bridged.

Limits

  • It recalls what a question asks for in similar words. The vote compares the question's attention queries with stored keys; a question that shares little with the note that answers it can miss it.
  • Two-step questions are the weak case, as on the other V15 models: the second fact is recalled when its vote reaches half of the first one's (recall_ratio), or 35% of it (linked_ratio) once a sleep has bridged the two events. Bridges exist only where the model, asked during a sleep which person, thing or place two notes both name, gave a name written in both.
  • A corrected fact is retired only by a sleep. Until the server has been idle long enough to sleep (VLLM_V15_SLEEP_AFTER, 10 minutes by default), the old and the new value are both recalled; during the sleep the model is asked, in two wordings, whether the later note changes the earlier one, and only two yeses retire the old one, which is then recalled only together with the new one, before it. On the 0.8B model the core was measured on, this judge recognised few corrections; it has not been measured on this model. Recalled events carry no time stamp in this image.
  • The sleep uses the GPU while the server is idle. A cycle replays up to 64 events (and asks, of each one with a question mark, whether it only asks), judges up to 128 pairs (two questions each), dreams up to 16 questions and asks up to 64 pairs which name they share, all through the server itself. A user's request is served beside the sleep's request in flight and stops the cycle at its next step; the next idle stretch takes up the rest.
  • Decoding rows beside a prefill read no memory in that step. vLLM runs prefill chunks and decoding requests in one batch; a decoding row in such a batch attends only its own context for that step.
  • A prefill with memory runs without a CUDA graph (it votes and recalls on the host); with this profile prefills above 64 tokens do anyway. Decode and MTP verify keep their CUDA graphs, the merge captured in them.
  • Requests beyond VLLM_V15_MAX_SLOTS (4) run without memory (requests that have a slot keep theirs in the same steps), and requests given as prompt embeddings do too.
  • A prompt the engine has read before (a regenerated reply) is answered from its checkpoint without a forward; it recalls what its first reading recalled when that was one of the last 64 requests, and otherwise only from the vote at its 32nd generated token.
  • Pictures and videos of a prompt are stored as their placeholder tokens; they are not identified, so a picture is not recognized again later.
  • No separation between users. Every request reads and writes the same store, which keeps what the prompts contained (token ids, attention keys and values). Run one server with its own /memory for each user or tenant whose memories must stay apart; delete the directory to forget.
  • GPU memory. The memory's buffers and index are taken from the KV cache at start-up (see Serve).
  • Host memory under Docker Desktop (WSL 2). The virtual machine's page cache, the 27 GB PLE table the profile keeps in host memory and the GPU memory all count against the host's commit limit; on the machine this was measured on the start failed until the page cache was dropped while the weights loaded. Keep memory in .wslconfig well below the host's RAM plus page file minus the GPU memory.
  • The recall thresholds (recall_ratio, recall_floor, linked_ratio) were set on Qwen3.5-0.8B and not re-tuned for this model.
  • The trained weights are untouched, so everything the original card says about the model itself still holds, including its own limitations.

Credits and license

  • The model is DevelopingDad/Qwen3.8-Flash-Next-FP4-Local-Abliterated (from nvidia/Qwen3.8-Flash-Next-FP4-Local, from Qwen3.8-Flash-Next); its card follows below, unchanged, and its licenses apply to this repository (the NVIDIA Open Model Agreement in LICENSE, the Qwen Community License in upstream/, see NOTICE).
  • The serving engine is vLLM (Apache-2.0) as built by local-inference-lab with b12x kernels in FlashInfer (Apache-2.0). vllm-v15-patch/ changes five of the image's files (each marked as changed, the originals in vllm-v15-patch/pristine/) and adds the memory, under Apache-2.0 (vllm-v15-patch/LICENSE); the image it builds is not an official image of vLLM, FlashInfer or local-inference-lab.
  • Conversion, memory code and image patch: LTM-Lab / lovewin10.

Original model card (DevelopingDad/Qwen3.8-Flash-Next-FP4-Local-Abliterated)

Qwen3.8-Flash-Next FP4-Local — Abliterated

Fewer refusals, the same compact weight formats, and similar single-Spark decode speed to NVIDIA's FP4-Local base.

Base
Original
Method
Runs on
License

Quick start · Evaluation · Two-Spark recipe · Method

A standalone derivative of NVIDIA's Qwen3.8-Flash-Next-FP4-Local, modified through abliteration: a weight edit that reduces the model's learned refusal behavior. The edit is baked into the checkpoint, so no adapter or runtime hook is needed.

The weights occupy approximately 96 GiB on disk and run on one DGX Spark using the supplied launcher. A separate two-Spark serving recipe is linked below.

[!IMPORTANT]
This checkpoint keeps NVIDIA's FP4-Local storage format, including its compressed NVFP4 scales. Like the base, it requires a compatible B12X vLLM container. Use the pinned image in serve-spark.sh; a standard Transformers or stock vLLM loading example is not a supported path for this checkpoint.

Highlights

  • Substantially reduced refusals: 0 of 100 development and 6 of 100 holdout prompts were detected as refusals, against 99 and 100 for the NVIDIA base.
  • Base-compatible weight layout: every tensor retains the base checkpoint's name, dtype and shape. Routed experts remain NVFP4; attention and shared-expert projections remain MXFP8.
  • Similar single-Spark speed: median decode rates of 38.0 tok/s for prose and 54.3 tok/s for code, against 39.4 and 50.7 for the base under the same serving profile.
  • Functional checks passed: image input, tool calling, thinking controls and MTP speculative decoding passed the checks described below.
  • Self-contained: download the repository and run the included DGX Spark launcher.

These results describe the MXFP8 release dated October 22, 2026. They are bounded measurements, not a claim that all capabilities are unchanged.

Model overview

Original model Qwen/Qwen3.8-Flash-Next by Qwen
Quantized base nvidia/Qwen3.8-Flash-Next-FP4-Local by NVIDIA, revision b9ce18a
Parameters 125B language-model parameters, 6B active, plus a 51B n-gram embedding table and a 4B MTP layer, as described in NVIDIA's card
Input / output Text and images in, text out
Weight storage Approximately 96 GiB / 102 GB across 21 shards
Precision Mixed precision: NVFP4 routed experts, MXFP8 attention and shared-expert projections, and other tensors in their original base formats
What changed Refusal-direction ablation in the 48 attention output projections and the routed/shared expert down projections
Single-Spark context Configured for 262,144 tokens; retrieval checks covered approximately 8K and 30K tokens
Tested hardware DGX Spark, GB10, 128 GB unified memory; separate single-node and two-node profiles

Weight storage is the size on disk, not the complete runtime memory requirement. The single-Spark launcher uses disk-backed n-gram embeddings and a 104 GiB container memory cap.

Quick start (DGX Spark)

You need:

  • One DGX Spark with Docker and NVIDIA GPU support.
  • The Hugging Face CLI, hf. See the installation guide.
  • Space for approximately 102 GB of model files, plus the Docker image and compilation caches.
  • Enough free unified memory to run the 104 GiB container. Avoid running another large model alongside it.

Download the complete repository, preserving the tensors/ directory, then start the server:

hf download DevelopingDad/Qwen3.8-Flash-Next-FP4-Local-Abliterated \
  --local-dir ./qwen-abliterated

bash ./qwen-abliterated/serve-spark.sh ./qwen-abliterated

The launcher uses this pinned runtime image:

eugr/spark-vllm-b12x@sha256:03a956544e9f9b4b7ba91690469fbe7848bdbef29a7d2e0f5788368467bb3595

First start: compilation and autotuning took about 23 minutes on the tested Spark. Later starts reused the cache and took about 5 minutes. Startup time varies by host and cache state.

The launcher stays attached to the server logs. In another terminal, check readiness:

curl --fail http://127.0.0.1:8000/health

Once that succeeds, the OpenAI-compatible API is available at http://127.0.0.1:8000/v1, with model ID qwen-trial117.

Send a request

This example disables thinking and requests a direct answer:

curl --fail http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen-trial117",
    "messages": [
      {
        "role": "user",
        "content": "Explain how a heat pump works in three sentences."
      }
    ],
    "max_tokens": 256,
    "chat_template_kwargs": {
      "enable_thinking": false
    }
  }'

For Python, install the client in your Python environment:

python3 -m pip install openai

This example enables thinking explicitly:

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="none",
)

reply = client.chat.completions.create(
    model="qwen-trial117",
    messages=[
        {"role": "user", "content": "Write a haiku about unified memory."}
    ],
    max_tokens=2048,
    extra_body={
        "chat_template_kwargs": {"enable_thinking": True}
    },
)

message = reply.choices[0].message
reasoning = getattr(message, "reasoning", None)
if reasoning:
    print(reasoning)
print(message.content)
  • Thinking is on by default. Set chat_template_kwargs.enable_thinking per request. Reasoning is returned in the message's reasoning field.
  • Tools use the OpenAI-compatible tools field, with the qwen3_xml parser.
  • Sampling defaults come from generation_config.json: temperature 1.0, top-p 0.95 and top-k 20. The greedy evaluation settings below are separate from these defaults.

To stop the server from another terminal:

docker stop -t 90 qwen-trial117

Rerun the launcher to start it again; compiled caches persist.

Serving profile

The launcher follows NVIDIA's published DGX Spark profile for the base:

Setting Default
Tensor parallelism 1
Maximum context 262,144 tokens
Active sequences Up to 8
KV cache FP8
Prefix caching / chunked prefill Enabled
Speculative decoding MTP, 4 draft tokens
Kernels B12X MoE, linear and GDN
N-gram embeddings / PLE Disk-backed
Container memory cap 104 GiB

Eight active sequences do not mean eight simultaneous maximum-length contexts; available KV cache limits the total.

Variable Default Effect
PORT 8000 API port
HOST 127.0.0.1 Bind address; use an authenticating proxy before exposing the API beyond the host
MTP_TOKENS 4 Speculative draft tokens; 0 disables MTP
CACHE_DIR ~/.cache/qwen-trial117 Compiled kernels and autotune results

Evaluation

The following results cover the October 22, 2026 MXFP8 release on one DGX Spark, using serve-spark.sh unless otherwise stated. NVIDIA's base checkpoint was served with the same profile.

The base revision, runtime image, frozen intervention inputs and validation summary are recorded in provenance.json. File hashes and sizes are recorded in SHA256SUMS and release-files.json.

Refusals and drift

Refusal tests used 100 development and 100 holdout prompts from mlabonne/harmful_behaviors. Answers were greedy, seed 3741, thinking off and capped at 100 tokens. A refusal regex classified the responses.

Checkpoint Development refusals Holdout refusals
NVIDIA base 99 / 100 100 / 100
This model 0 / 100 6 / 100

These are detected refusals in short responses. They do not measure answer correctness, usefulness or universal compliance. Raw responses to refusal-test prompts are not published.

Distribution drift was measured separately on benign prompts from mlabonne/harmless_alpaca, using 100 prompts per split:

Metric, this model versus NVIDIA base Development Holdout
First-token KL, nats 0.207 0.219
First-token top-1 agreement 81% 84%

KL measures a difference between probability distributions. Here it uses the first answer token and renormalized top-20 candidates. It is not a percentage, a full-vocabulary KL measurement or a general capability score.

Capabilities

  • Functional checks: 11 of 11 passed. Model listing, streamed usage, thinking on and off, automatic and forced tool calls, a tool-result follow-up returning exact JSON, image colour recognition, and exact retrieval of three planted markers from documents of about 8,000 and 30,000 tokens.
  • Vision: 10 of 10 solid-colour images named correctly. This is a basic image-input check, not a broad vision benchmark.
  • Speculative decoding: observed MTP acceptance was comparable to the base. A greedy audit found no emitted-token mismatches against the target model's top choice on the audited output.
  • Writing: 13 of 20 editing and composition cases passed all scoring dimensions, against 14 for the base. An AI reviewer scored the fixture; it was not human-adjudicated. The two changed cases also varied between runs of the other checkpoints.

Two writing behaviors repeated across runs: short arithmetic questions could trigger step-by-step explanations that exhausted a tight output budget, and requests to tighten a long note received lighter edits than the base.

These checks establish specific working paths. They do not establish unchanged overall capability.

Speed

Single request, greedy decoding, thinking off, 512 output tokens, MTP 4. Decode rates are medians of six probes over two rounds; ranges are shown in parentheses.

Metric This model NVIDIA base
Prose decode, tok/s 38.0 (37.7–39.6) 39.4 (39.1–40.9)
Code decode, tok/s 54.3 (48.5–56.0) 50.7 (49.3–52.4)
Speculative decode step 67.7–67.9 ms 66.8–67.2 ms
KV cache capacity 947,989 tokens 902,668 tokens
Time to first token, short prompt 0.05–0.27 s —
Time to first token, ~30,000-token prompt, cold 12.7 s —

Decode rate measures generation after the first content token; it excludes prefill. Speed depends strongly on content because speculative decoding accepts more draft tokens on predictable text such as code.

On a separately tuned single-Spark stack with MTP 2, based on diffbot's DGX Spark recipe, prose reached 45.3–45.7 tok/s against 46.1–47.1 for the base. That stack is not included in this repository, and those figures do not describe the supplied launcher's defaults.

Two DGX Sparks

For two-node serving, see XtraSaltyDev's TP=2 recipe and evaluation. It serves this checkpoint without modifying the weights.

The qualified profile uses 262K context, 16 active sequences, BF16 KV cache, MTP4 and disk-backed PLE, with a 104 GiB container cap on each Spark. It requires working RoCE networking and the model files on both nodes.

Measured on October 22, 2026:

Workload Original eight-slot profile Qualified sixteen-slot profile
16 concurrent short requests: aggregate output 229.0 tok/s 313.4 tok/s (+37%)
Same workload: p95 first token 11.89 s 0.77 s
16 concurrent ~8K-input requests: aggregate output 38.64 tok/s 40.98 tok/s (+6%)

Both columns use the same abliterated checkpoint and runtime image. This comparison measures serving-profile changes, not performance against NVIDIA's unmodified model.

The short workload used 62–64 input tokens and 256 forced output tokens, greedy decoding and thinking off. Aggregate throughput includes queue and prefill time across the request wave; it is not a single user's decode rate. The samples are small and do not establish production latency guarantees. Serial prose decode was about 8% slower than the eight-slot reference.

The selected profile also passed 10/10 strict JSON retrieval checks at 261,623 input tokens. This result belongs to the two-Spark BF16-KV profile; it does not extend the single-Spark FP8-KV validation above. Sixteen scheduler slots do not imply sixteen simultaneous 262K contexts.

See the GitHub repository for setup, full results and tradeoffs, and the pinned checkpoint revision and raw evidence.

How it was made

  1. Find the refusal direction. The method follows Arditi et al. (2024) and the parameterization of Heretic, reimplemented for this checkpoint's native vLLM backend. Per-layer directions come from attention-output activations on harmful and harmless prompt sets. An Optuna TPE search balanced refusals against KL, and trial 117 was selected.

  2. Bake the intervention into the weights. The selected projection was applied to the 48 attention output projections and the routed/shared expert down projections. Expert weights were re-rounded in NVFP4 and MXFP8 while preserving their scale encodings.

  3. Restore attention to MXFP8. The intermediate bake stored the edited attention output weights in BF16. Constrained rounding re-encoded them into the base's MXFP8 format while preserving the intended directional edit.

The selected trial uses the accepted continuation recalibration. Its parameters, directions and row norms are included in bake-selection.json, frozen-directions.json and frozen-row-norms.json.

Direction-preserving re-encode

Plain MXFP8 rounding of the edited attention weights would introduce error along the removed direction equal to about a tenth of the intervention's size.

The re-encode script instead rounds each weight to one of its two nearest E4M3 values, using the base's scale rule of one E8M0 exponent per 32 inputs. A greedy pass chooses roundings that cancel error along the removed direction while minimizing added squared error.

Across the 48 attention output projections:

  • Error along the removed direction is at most 0.002% of the intervention.
  • Overall relative weight error is 1.47–1.93%.
  • The fraction of the direction removed matches the BF16 intermediate to within 0.00002.

Per-layer measurements are in reencode/reencode-report.json. These are weight-space measurements, separate from the behavioral tests above.

The tokenizer, chat template, vision encoder and built-in MTP layer are NVIDIA's. The model configuration is unchanged apart from a standalone_provenance metadata entry. This is an edited derivative of NVIDIA's quantized checkpoint, not a repeat of its quantization-aware distillation.

Files

File Contents
tensors/, model.safetensors.index.json Weights: 21 shards in the base layout
config.json, tokenizer, chat_template.jinja, preprocessor configs Base configuration and processing files, with the provenance addition noted above
serve-spark.sh Single-Spark launcher
bake-selection.json, frozen-directions.json, frozen-row-norms.json Frozen trial 117 intervention inputs
reencode/ MXFP8 re-encode script and per-layer report
provenance.json Build information and validation summary
SHA256SUMS, release-files.json Release file hashes and sizes
LICENSE, NOTICE, upstream/ Governing terms, notices and NVIDIA's original card

Limitations

  • Narrow evaluation: broad capability benchmarks and video input were not evaluated for this derivative. The vision checks were limited to solid colours.
  • Profile-specific evidence: single-Spark retrieval checks reached about 30K tokens. The separate two-Spark profile passed the near-262K checks described above. Neither proves reliable performance on every task at those lengths.
  • Original targets not met: the project aimed for zero refusals and a KL ceiling of 0.20 and did not meet its original qualification targets. The top-20 KL reported here is distinct from the original full-vocabulary metric.
  • Refusals remain: 6 of 100 holdout prompts were detected as refusals.
  • Hardware and runtime scope: validation used DGX Spark and pinned B12X runtimes. Other hardware and runtime versions are not qualified here.
  • Outputs can be wrong or inconsistent: reduced refusal behavior does not establish correctness, factual reliability or preserved general capability.

Responsible use

Abliteration reduces learned refusal behavior. The model may attempt requests the original declines and can produce harmful, offensive, false or illegal content.

  • You are responsible for how you use it and for complying with applicable law and the governing licenses.
  • Add your own safeguards before deploying it to end users.
  • Do not use it for activities prohibited by the Qwen or NVIDIA terms.

There is no warranty of any kind.

License

This checkpoint is derived from NVIDIA's quantized checkpoint and Qwen's original model. Their governing terms continue to apply; this release does not relicense the base weights.

  • NVIDIA Open Model Agreement: the governing download terms for NVIDIA's checkpoint. A copy is included in LICENSE.
  • Qwen Community License 1.0: additional terms for the original model, included in upstream/QWEN-LICENSE.

Licensed by NVIDIA Corporation under the NVIDIA Open Model Agreement.

Read the full license texts before use or redistribution, including their commercial-use conditions. Runtime containers and external software have their own licenses.

Citation

@misc{developingdad2026qwen38abliterated,
  title        = {Qwen3.8-Flash-Next FP4-Local -- Abliterated},
  author       = {DevelopingDad},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/DevelopingDad/Qwen3.8-Flash-Next-FP4-Local-Abliterated}}
}

@article{arditi2024refusal,
  title   = {Refusal in Language Models Is Mediated by a Single Direction},
  author  = {Arditi, Andy and Obeso, Oscar and Syed, Aaquib and Paleka, Daniel and Panickssery, Nina and Gurnee, Wes and Nanda, Neel},
  journal = {arXiv preprint arXiv:2406.11717},
  year    = {2024}
}

@misc{heretic,
  author       = {Weidmann, Philipp Emanuel},
  title        = {Heretic: Fully automatic censorship removal for language models},
  year         = {2025},
  publisher    = {GitHub},
  journal      = {GitHub repository},
  howpublished = {\url{https://github.com/p-e-w/heretic}}
}

Credits

This release stands on other people's work.

Model

  • Qwen: Qwen3.8-Flash-Next, the original model, under the Qwen Community License 1.0.
  • NVIDIA: the FP4-Local checkpoint, quantized with Model Optimizer (Apache-2.0), under the NVIDIA Open Model Agreement. NVIDIA's published DGX Spark profile is the basis of serve-spark.sh.

Method

Software and serving

  • vLLM (Apache-2.0): the serving engine.
  • FlashInfer (Apache-2.0) and B12X by Local Inference Lab (Apache-2.0): the kernels.
  • eugr/spark-vllm-docker (MIT): the DGX Spark vLLM image used for building and serving.
  • Optuna (MIT), PyTorch (BSD-3-Clause) and safetensors (Apache-2.0).
  • diffbot: the tuned-stack recipe behind the optional single-Spark speed figure.
  • XtraSaltyDev: the separate two-Spark serving recipe, evaluation harnesses and deployment results.

External projects retain their own licenses. This release is not endorsed by Qwen, NVIDIA or any of the projects above.

README history 3 versions

The author's README evolved over time. Click a version to see its content at that point.

  1. 2026-11-17 V15 episodic memory conversion: configuration, code, card, checksums and the ... bc85096 38.3 KB
    Loading...
  2. 2026-11-16 V15 episodic memory conversion: configuration, code, card, checksums and the ... 47fc6a6 38.1 KB
    Loading...
  3. 2026-11-16 V15 episodic memory conversion: configuration, code, card, checksums and the ... 66f5ff1 37.7 KB
    Loading...
Catalog is the map. Apps are the tools.

Run models on your own machine, not in the cloud.

Every model page has an "Open in Abliteration" button that hands the model directly to the first-party desktop client, at the quantization your rig can actually run. No API keys, no subscription, no prompt leakage.

Open in Abliteration