license: other
license_name: nvidia-open-model-agreement
license_link: LICENSE
pipeline_tag: image-text-to-text
datasets:
- mlabonne/harmful_behaviors
- mlabonne/harmless_alpaca
tags: - v15-episodic-memory
- qwen3.8
- qwen3.8-flash-next
- abliterated
- heretic
- nvfp4
- mxfp8
- modelopt
- mtp
- dgx-spark
- vllm
base_model: - DevelopingDad/Qwen3.8-Flash-Next-FP4-Local-Abliterated
Qwen3.8-Flash-Next-FP4-Local-Abliterated-v15
繁體中文摘要:這是 DevelopingDad/Qwen3.8-Flash-Next-FP4-Local-Abliterated 的 V15 轉換版。權重與原模型逐位元相同(FP4-Local 格式不變),多的是記憶的設定、兩個程式檔,以及一份鏡像補丁 vllm-v15-patch/:在 local-inference-lab 的 vLLM 鏡像(ghcr.io/local-inference-lab/vllm:release-10152026,能跑 FP4-Local 格式的 b12x 版 vLLM)上加入記憶;建好的鏡像是 lovewin10/vllm-v15:release-10152026-v15。用這個鏡像服務時,模型帶有跨請求保存的情節記憶:在一段對話裡說過的事,之後在另一段全新的對話裡問得到。V15 與 V14 相比拿掉了世界模型,多了睡眠:伺服器閒置時,用被服務的模型自己整理記憶庫(重播、合併重複、判斷新事實是否取代舊事實、建立相關事件之間的橋),不訓練、不寫入新內容。CUDA graph、前綴快取、MTP 推測解碼與非同步排程照常開著。實測結果與已知的限制寫在下面。
DevelopingDad/Qwen3.8-Flash-Next-FP4-Local-Abliterated converted to the V15 episodic memory architecture (Qwen4ExpV15ForConditionalGeneration).
- The weights are the original's, byte for byte, in its FP4-Local format (ModelOpt mixed precision, NVFP4 experts with compressed scales). Nothing was trained, merged or re-quantized here; the memory adds no parameters.
- Added: the architecture name, an
auto_mapand the memory options (qwen4_exp_v15_memory_config) inconfig.json, two code files (configuration_qwen4_exp_v15.py,modeling_qwen4_exp_v15.py) that hold the memory, and the image patchvllm-v15-patch/.SHA256SUMSlists this repository's files; the weights' lines are the original's. - Served by the image
lovewin10/vllm-v15:release-10152026-v15(built fromvllm-v15-patch/), the model has a memory that persists across requests: what was said in one chat is answered in a later, separate chat. The store is on disk and survives restarts. - The image is
ghcr.io/local-inference-lab/vllm:release-10152026(the vLLM that serves this checkpoint's format with b12x kernels) with the memory added; it is not an official vLLM or local-inference-lab image.
How the memory works
While a prompt is read, the keys and values of its 12 indexed-attention (QSA) layers are cut into events of up to 128 tokens and written to a store on disk, each with a low-rank engram of the state it leaves in the 36 Gated DeltaNet layers. A later prompt is read once to see what it asks for; the stored events whose keys would take its attention are recalled by one vote for all layers and heads (with one hop along the bridges a sleep confirmed, and an event a sleep found superseded replaced by the one that supersedes it); their engrams are composed into the request's recurrent state and the prompt is read again with the recalled keys and values merged into the attention (by the local attention's log-sum-exp, which the patched b12x QSA kernels return). Generated tokens attend the recalled events too, inside the decode CUDA graphs. Nothing is retrieved as text and nothing is added to the prompt.
While no user request has come for 10 minutes (VLLM_V15_SLEEP_AFTER), the server sleeps: a thread of the engine process replays the events no sleep has seen yet through the server's own API with the memory read-only, merges repeated ones, takes out of recall a note the model says twice only asks something, asks the model in two wordings whether a later note changes an earlier one (two yeses retire the earlier one: it is then recalled only with the later one, before it), dreams a question each event answers and asks which person, thing or place two events it calls up together both name (a name written in both is a bridge), and downscales the strength of what was not used. It changes only how the store is organised: no training, no new content, no event deleted. A user's request stops a cycle at its next step. Every cycle writes one V15 sleep: line to the log.
Serve
Build the image from this repository's folder (the build pulls ghcr.io/local-inference-lab/vllm:release-10152026, about 44 GB, and stops if any file the patch replaces differs from the original it was made from), or pull it when it is on Docker Hub:
docker build -t lovewin10/vllm-v15:release-10152026-v15 vllm-v15-patch
# or: docker pull lovewin10/vllm-v15:release-10152026-v15
It starts as NVIDIA's command for one RTX PRO 6000 Blackwell starts its base (PROFILE=qwen38-flash-next); only the image and the model folder change, and a folder for the store is mounted at /memory. On one 96 GB GPU (under Docker Desktop, where the port is published instead of --network host):
docker run -d --name flash-next-v15 --init --gpus device=0 `
--ipc host --shm-size 32g --stop-timeout 30 `
--ulimit memlock=-1 --ulimit stack=67108864:67108864 -p 127.0.0.1:8000:8000 `
-e PROFILE=qwen38-flash-next -e MODEL=/models/qwen -e PORT=8000 `
-v <this repository>:/models/qwen:ro -v flash-next-cache:/cache -v <a folder for the store>:/memory `
lovewin10/vllm-v15:release-10152026-v15
The profile carries the rest of NVIDIA's settings: a 262,144-token context, 16 sequences, FP8 KV cache, MTP with 3 draft tokens, b12x kernels, GPU memory utilization 0.96, the qwen3 reasoning parser and the qwen3_xml tool parser. The model is served as Qwen3.8-Flash-Next, with the sampling defaults of generation_config.json (temperature 1.0, top-p 0.95, top-k 20); thinking is on unless a request sets chat_template_kwargs.enable_thinking to false.
VLLM_V15_MEMORY=0serves the folder as the plain checkpoint (no memory), exactly as the base image does.VLLM_V15_SLEEP=0keeps the memory and turns its sleep off;VLLM_V15_SLEEP_AFTER(default 600) is the idle time in seconds before a cycle starts. The sleep calls the server athttp://127.0.0.1:$PORT; setVLLM_V15_SELF_URLwhen it listens elsewhere, and give an API key asVLLM_API_KEY(not--api-key), which the sleep sends as well.VLLM_V15_MAX_SLOTS(default 4): requests that read the memory at once; each holds about 100 MiB of decode buffers taken from the KV cache. Other requests of the same step run without memory, the ones with a slot keep theirs.VLLM_V15_INDEX_GB(unset by default): the store's index rows are reserved at start-up for the payloads it holds and 8,192 more (about 0.53 GiB for a new store, at most 2 GiB), and the log warns when they are 80% full (a restart sizes them anew); a number pins the size. Beyond the rows a vote reads the rest back from the store.VLLM_V15_WORK_GB(default 1.25): GPU memory held back from the KV cache for the memory's work the vLLM profile does not see (a prefill's recurrent states, the keys of recent votes, the far contexts of the requests that recall).VLLM_V15_KV_CACHE_GB(default 0.25): GPU memory for the decoded keys of recently recalled events.VLLM_V15_TRACE=1logs every recall decision.- With the memory on the KV cache is smaller (669,427 tokens instead of 766,241 with the defaults above, measured). The GPU still shows nearly full, as with the base image: the profile hands everything below
--gpu-memory-utilization 0.96to the KV cache. A lower-e GPU_MEMORY_UTILIZATION(or a fixed-e KV_CACHE_MEMORY_BYTES=..., the base image's options) leaves GPU memory free, all of it taken from the KV cache: about 0.96 GiB, some 52,000 tokens, for every 0.01. Below about 0.90 one 262,144-token request no longer fits and vLLM stops after loading the weights, so add-e MAX_MODEL_LEN=-1as well (vLLM serves the longest context that fits and logs it asAuto-fit max_model_len) or a smallerMAX_MODEL_LEN. These figures are worked out from the V14 image's start-up log at 0.96, not measured at the lower settings. - Other images do not know the V15 architecture name, including the DGX Spark image of the original's
serve-spark.sh: there, serve the original repository (or setarchitecturesinconfig.jsonback toQwen4ExpForConditionalGeneration).vllm-v15-patch/README.mdlists what the patch changes.
Measured
One RTX PRO 6000 Blackwell (96 GB) under Docker Desktop, 2026-11-16, with the published image, the profile above, an empty store and -e VLLM_V15_SLEEP_AFTER=20 (so that the check sees a sleep); every statement and every question a chat of its own, temperature 0, thinking off. A handful of questions per check: a functional check, not a benchmark.
| Memory on (V15) | Memory off (the base image, measured with the V14 image's memory off) | |
|---|---|---|
| A fact told in one chat, asked in a new chat (6 facts) | 6 / 6 | 0 / 6 |
| Unrelated questions asked afterwards (4) | 4 / 4 | 4 / 4 |
| The six facts asked again after one sleep | 6 / 6 | |
| Decode speed, three answers of 266 to 400 tokens | 119.8 tokens/s | 122.4 tokens/s |
| First token of those three short prompts | 1.03 s | 0.15 s |
| KV cache | 669,427 tokens | 766,241 tokens |
| Ready after (weights on an SSD) | 1779 s |
The sleep cycle took 87.8 s (49 events replayed, 3 told twice merged, no correction among these facts and none made, 3 bridges); no traceback was logged. The memory's index takes 0.53 GiB of the device at start-up. Against the V14 image on this checkpoint (99.0 tokens/s, first token 1.06 s, KV cache 656,518 tokens), V15 decodes faster: the event V14's world model expected next is no longer attended.
The V15 memory core was also measured on Qwen3.5-0.8B with Hugging Face Transformers, one sleep between telling and asking against no sleep: single facts 9/10 both ways, facts among 48 other notes 7/10 both ways, among 248 6/10 both ways, corrected facts 1/6 without and 2/6 with the sleep, two-step questions 5/6 without and 6/6 with it, long prompts 3/3, unrelated questions 7/8. Corrections and two-step questions were measured on the 27B FP8 conversion with the SGLang image (its card): after a sleep 6 of 6 corrected facts answered with the new value, and every pair a two-step question needs bridged.
Limits
- It recalls what a question asks for in similar words. The vote compares the question's attention queries with stored keys; a question that shares little with the note that answers it can miss it.
- Two-step questions are the weak case, as on the other V15 models: the second fact is recalled when its vote reaches half of the first one's (
recall_ratio), or 35% of it (linked_ratio) once a sleep has bridged the two events. Bridges exist only where the model, asked during a sleep which person, thing or place two notes both name, gave a name written in both. - A corrected fact is retired only by a sleep. Until the server has been idle long enough to sleep (
VLLM_V15_SLEEP_AFTER, 10 minutes by default), the old and the new value are both recalled; during the sleep the model is asked, in two wordings, whether the later note changes the earlier one, and only two yeses retire the old one, which is then recalled only together with the new one, before it. On the 0.8B model the core was measured on, this judge recognised few corrections; it has not been measured on this model. Recalled events carry no time stamp in this image. - The sleep uses the GPU while the server is idle. A cycle replays up to 64 events (and asks, of each one with a question mark, whether it only asks), judges up to 128 pairs (two questions each), dreams up to 16 questions and asks up to 64 pairs which name they share, all through the server itself. A user's request is served beside the sleep's request in flight and stops the cycle at its next step; the next idle stretch takes up the rest.
- Decoding rows beside a prefill read no memory in that step. vLLM runs prefill chunks and decoding requests in one batch; a decoding row in such a batch attends only its own context for that step.
- A prefill with memory runs without a CUDA graph (it votes and recalls on the host); with this profile prefills above 64 tokens do anyway. Decode and MTP verify keep their CUDA graphs, the merge captured in them.
- Requests beyond
VLLM_V15_MAX_SLOTS(4) run without memory (requests that have a slot keep theirs in the same steps), and requests given as prompt embeddings do too. - A prompt the engine has read before (a regenerated reply) is answered from its checkpoint without a forward; it recalls what its first reading recalled when that was one of the last 64 requests, and otherwise only from the vote at its 32nd generated token.
- Pictures and videos of a prompt are stored as their placeholder tokens; they are not identified, so a picture is not recognized again later.
- No separation between users. Every request reads and writes the same store, which keeps what the prompts contained (token ids, attention keys and values). Run one server with its own
/memoryfor each user or tenant whose memories must stay apart; delete the directory to forget. - GPU memory. The memory's buffers and index are taken from the KV cache at start-up (see Serve).
- Host memory under Docker Desktop (WSL 2). The virtual machine's page cache, the 27 GB PLE table the profile keeps in host memory and the GPU memory all count against the host's commit limit; on the machine this was measured on the start failed until the page cache was dropped while the weights loaded. Keep
memoryin.wslconfigwell below the host's RAM plus page file minus the GPU memory. - The recall thresholds (
recall_ratio,recall_floor,linked_ratio) were set on Qwen3.5-0.8B and not re-tuned for this model. - The trained weights are untouched, so everything the original card says about the model itself still holds, including its own limitations.
Credits and license
- The model is DevelopingDad/Qwen3.8-Flash-Next-FP4-Local-Abliterated (from nvidia/Qwen3.8-Flash-Next-FP4-Local, from Qwen3.8-Flash-Next); its card follows below, unchanged, and its licenses apply to this repository (the NVIDIA Open Model Agreement in
LICENSE, the Qwen Community License inupstream/, seeNOTICE). - The serving engine is vLLM (Apache-2.0) as built by local-inference-lab with b12x kernels in FlashInfer (Apache-2.0).
vllm-v15-patch/changes five of the image's files (each marked as changed, the originals invllm-v15-patch/pristine/) and adds the memory, under Apache-2.0 (vllm-v15-patch/LICENSE); the image it builds is not an official image of vLLM, FlashInfer or local-inference-lab. - Conversion, memory code and image patch: LTM-Lab / lovewin10.
Original model card (DevelopingDad/Qwen3.8-Flash-Next-FP4-Local-Abliterated)
Qwen3.8-Flash-Next FP4-Local — Abliterated
Fewer refusals, the same compact weight formats, and similar single-Spark decode speed to NVIDIA's FP4-Local base.
A standalone derivative of NVIDIA's Qwen3.8-Flash-Next-FP4-Local, modified through abliteration: a weight edit that reduces the model's learned refusal behavior. The edit is baked into the checkpoint, so no adapter or runtime hook is needed.
The weights occupy approximately 96 GiB on disk and run on one DGX Spark using the supplied launcher. A separate two-Spark serving recipe is linked below.
[!IMPORTANT]
This checkpoint keeps NVIDIA's FP4-Local storage format, including its compressed NVFP4 scales. Like the base, it requires a compatible B12X vLLM container. Use the pinned image inserve-spark.sh; a standard Transformers or stock vLLM loading example is not a supported path for this checkpoint.
Highlights
- Substantially reduced refusals: 0 of 100 development and 6 of 100 holdout prompts were detected as refusals, against 99 and 100 for the NVIDIA base.
- Base-compatible weight layout: every tensor retains the base checkpoint's name, dtype and shape. Routed experts remain NVFP4; attention and shared-expert projections remain MXFP8.
- Similar single-Spark speed: median decode rates of 38.0 tok/s for prose and 54.3 tok/s for code, against 39.4 and 50.7 for the base under the same serving profile.
- Functional checks passed: image input, tool calling, thinking controls and MTP speculative decoding passed the checks described below.
- Self-contained: download the repository and run the included DGX Spark launcher.
These results describe the MXFP8 release dated October 22, 2026. They are bounded measurements, not a claim that all capabilities are unchanged.
Model overview
| Original model | Qwen/Qwen3.8-Flash-Next by Qwen |
| Quantized base | nvidia/Qwen3.8-Flash-Next-FP4-Local by NVIDIA, revision b9ce18a |
| Parameters | 125B language-model parameters, 6B active, plus a 51B n-gram embedding table and a 4B MTP layer, as described in NVIDIA's card |
| Input / output | Text and images in, text out |
| Weight storage | Approximately 96 GiB / 102 GB across 21 shards |
| Precision | Mixed precision: NVFP4 routed experts, MXFP8 attention and shared-expert projections, and other tensors in their original base formats |
| What changed | Refusal-direction ablation in the 48 attention output projections and the routed/shared expert down projections |
| Single-Spark context | Configured for 262,144 tokens; retrieval checks covered approximately 8K and 30K tokens |
| Tested hardware | DGX Spark, GB10, 128 GB unified memory; separate single-node and two-node profiles |
Weight storage is the size on disk, not the complete runtime memory requirement. The single-Spark launcher uses disk-backed n-gram embeddings and a 104 GiB container memory cap.
Quick start (DGX Spark)
You need:
- One DGX Spark with Docker and NVIDIA GPU support.
- The Hugging Face CLI,
hf. See the installation guide. - Space for approximately 102 GB of model files, plus the Docker image and compilation caches.
- Enough free unified memory to run the 104 GiB container. Avoid running another large model alongside it.
Download the complete repository, preserving the tensors/ directory, then start the server:
hf download DevelopingDad/Qwen3.8-Flash-Next-FP4-Local-Abliterated \
--local-dir ./qwen-abliterated
bash ./qwen-abliterated/serve-spark.sh ./qwen-abliterated
The launcher uses this pinned runtime image:
eugr/spark-vllm-b12x@sha256:03a956544e9f9b4b7ba91690469fbe7848bdbef29a7d2e0f5788368467bb3595
First start: compilation and autotuning took about 23 minutes on the tested Spark. Later starts reused the cache and took about 5 minutes. Startup time varies by host and cache state.
The launcher stays attached to the server logs. In another terminal, check readiness:
curl --fail http://127.0.0.1:8000/health
Once that succeeds, the OpenAI-compatible API is available at http://127.0.0.1:8000/v1, with model ID qwen-trial117.
Send a request
This example disables thinking and requests a direct answer:
curl --fail http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "qwen-trial117",
"messages": [
{
"role": "user",
"content": "Explain how a heat pump works in three sentences."
}
],
"max_tokens": 256,
"chat_template_kwargs": {
"enable_thinking": false
}
}'
For Python, install the client in your Python environment:
python3 -m pip install openai
This example enables thinking explicitly:
from openai import OpenAI
client = OpenAI(
base_url="http://127.0.0.1:8000/v1",
api_key="none",
)
reply = client.chat.completions.create(
model="qwen-trial117",
messages=[
{"role": "user", "content": "Write a haiku about unified memory."}
],
max_tokens=2048,
extra_body={
"chat_template_kwargs": {"enable_thinking": True}
},
)
message = reply.choices[0].message
reasoning = getattr(message, "reasoning", None)
if reasoning:
print(reasoning)
print(message.content)
- Thinking is on by default. Set
chat_template_kwargs.enable_thinkingper request. Reasoning is returned in the message'sreasoningfield. - Tools use the OpenAI-compatible
toolsfield, with theqwen3_xmlparser. - Sampling defaults come from
generation_config.json: temperature 1.0, top-p 0.95 and top-k 20. The greedy evaluation settings below are separate from these defaults.
To stop the server from another terminal:
docker stop -t 90 qwen-trial117
Rerun the launcher to start it again; compiled caches persist.
Serving profile
The launcher follows NVIDIA's published DGX Spark profile for the base:
| Setting | Default |
|---|---|
| Tensor parallelism | 1 |
| Maximum context | 262,144 tokens |
| Active sequences | Up to 8 |
| KV cache | FP8 |
| Prefix caching / chunked prefill | Enabled |
| Speculative decoding | MTP, 4 draft tokens |
| Kernels | B12X MoE, linear and GDN |
| N-gram embeddings / PLE | Disk-backed |
| Container memory cap | 104 GiB |
Eight active sequences do not mean eight simultaneous maximum-length contexts; available KV cache limits the total.
| Variable | Default | Effect |
|---|---|---|
PORT |
8000 |
API port |
HOST |
127.0.0.1 |
Bind address; use an authenticating proxy before exposing the API beyond the host |
MTP_TOKENS |
4 |
Speculative draft tokens; 0 disables MTP |
CACHE_DIR |
~/.cache/qwen-trial117 |
Compiled kernels and autotune results |
Evaluation
The following results cover the October 22, 2026 MXFP8 release on one DGX Spark, using serve-spark.sh unless otherwise stated. NVIDIA's base checkpoint was served with the same profile.
The base revision, runtime image, frozen intervention inputs and validation summary are recorded in provenance.json. File hashes and sizes are recorded in SHA256SUMS and release-files.json.
Refusals and drift
Refusal tests used 100 development and 100 holdout prompts from mlabonne/harmful_behaviors. Answers were greedy, seed 3741, thinking off and capped at 100 tokens. A refusal regex classified the responses.
| Checkpoint | Development refusals | Holdout refusals |
|---|---|---|
| NVIDIA base | 99 / 100 | 100 / 100 |
| This model | 0 / 100 | 6 / 100 |
These are detected refusals in short responses. They do not measure answer correctness, usefulness or universal compliance. Raw responses to refusal-test prompts are not published.
Distribution drift was measured separately on benign prompts from mlabonne/harmless_alpaca, using 100 prompts per split:
| Metric, this model versus NVIDIA base | Development | Holdout |
|---|---|---|
| First-token KL, nats | 0.207 | 0.219 |
| First-token top-1 agreement | 81% | 84% |
KL measures a difference between probability distributions. Here it uses the first answer token and renormalized top-20 candidates. It is not a percentage, a full-vocabulary KL measurement or a general capability score.
Capabilities
- Functional checks: 11 of 11 passed. Model listing, streamed usage, thinking on and off, automatic and forced tool calls, a tool-result follow-up returning exact JSON, image colour recognition, and exact retrieval of three planted markers from documents of about 8,000 and 30,000 tokens.
- Vision: 10 of 10 solid-colour images named correctly. This is a basic image-input check, not a broad vision benchmark.
- Speculative decoding: observed MTP acceptance was comparable to the base. A greedy audit found no emitted-token mismatches against the target model's top choice on the audited output.
- Writing: 13 of 20 editing and composition cases passed all scoring dimensions, against 14 for the base. An AI reviewer scored the fixture; it was not human-adjudicated. The two changed cases also varied between runs of the other checkpoints.
Two writing behaviors repeated across runs: short arithmetic questions could trigger step-by-step explanations that exhausted a tight output budget, and requests to tighten a long note received lighter edits than the base.
These checks establish specific working paths. They do not establish unchanged overall capability.
Speed
Single request, greedy decoding, thinking off, 512 output tokens, MTP 4. Decode rates are medians of six probes over two rounds; ranges are shown in parentheses.
| Metric | This model | NVIDIA base |
|---|---|---|
| Prose decode, tok/s | 38.0 (37.7–39.6) | 39.4 (39.1–40.9) |
| Code decode, tok/s | 54.3 (48.5–56.0) | 50.7 (49.3–52.4) |
| Speculative decode step | 67.7–67.9 ms | 66.8–67.2 ms |
| KV cache capacity | 947,989 tokens | 902,668 tokens |
| Time to first token, short prompt | 0.05–0.27 s | — |
| Time to first token, ~30,000-token prompt, cold | 12.7 s | — |
Decode rate measures generation after the first content token; it excludes prefill. Speed depends strongly on content because speculative decoding accepts more draft tokens on predictable text such as code.
On a separately tuned single-Spark stack with MTP 2, based on diffbot's DGX Spark recipe, prose reached 45.3–45.7 tok/s against 46.1–47.1 for the base. That stack is not included in this repository, and those figures do not describe the supplied launcher's defaults.
Two DGX Sparks
For two-node serving, see XtraSaltyDev's TP=2 recipe and evaluation. It serves this checkpoint without modifying the weights.
The qualified profile uses 262K context, 16 active sequences, BF16 KV cache, MTP4 and disk-backed PLE, with a 104 GiB container cap on each Spark. It requires working RoCE networking and the model files on both nodes.
Measured on October 22, 2026:
| Workload | Original eight-slot profile | Qualified sixteen-slot profile |
|---|---|---|
| 16 concurrent short requests: aggregate output | 229.0 tok/s | 313.4 tok/s (+37%) |
| Same workload: p95 first token | 11.89 s | 0.77 s |
| 16 concurrent ~8K-input requests: aggregate output | 38.64 tok/s | 40.98 tok/s (+6%) |
Both columns use the same abliterated checkpoint and runtime image. This comparison measures serving-profile changes, not performance against NVIDIA's unmodified model.
The short workload used 62–64 input tokens and 256 forced output tokens, greedy decoding and thinking off. Aggregate throughput includes queue and prefill time across the request wave; it is not a single user's decode rate. The samples are small and do not establish production latency guarantees. Serial prose decode was about 8% slower than the eight-slot reference.
The selected profile also passed 10/10 strict JSON retrieval checks at 261,623 input tokens. This result belongs to the two-Spark BF16-KV profile; it does not extend the single-Spark FP8-KV validation above. Sixteen scheduler slots do not imply sixteen simultaneous 262K contexts.
See the GitHub repository for setup, full results and tradeoffs, and the pinned checkpoint revision and raw evidence.
How it was made
Find the refusal direction. The method follows Arditi et al. (2024) and the parameterization of Heretic, reimplemented for this checkpoint's native vLLM backend. Per-layer directions come from attention-output activations on harmful and harmless prompt sets. An Optuna TPE search balanced refusals against KL, and trial 117 was selected.
Bake the intervention into the weights. The selected projection was applied to the 48 attention output projections and the routed/shared expert down projections. Expert weights were re-rounded in NVFP4 and MXFP8 while preserving their scale encodings.
Restore attention to MXFP8. The intermediate bake stored the edited attention output weights in BF16. Constrained rounding re-encoded them into the base's MXFP8 format while preserving the intended directional edit.
The selected trial uses the accepted continuation recalibration. Its parameters, directions and row norms are included in bake-selection.json, frozen-directions.json and frozen-row-norms.json.
Direction-preserving re-encode
Plain MXFP8 rounding of the edited attention weights would introduce error along the removed direction equal to about a tenth of the intervention's size.
The re-encode script instead rounds each weight to one of its two nearest E4M3 values, using the base's scale rule of one E8M0 exponent per 32 inputs. A greedy pass chooses roundings that cancel error along the removed direction while minimizing added squared error.
Across the 48 attention output projections:
- Error along the removed direction is at most 0.002% of the intervention.
- Overall relative weight error is 1.47–1.93%.
- The fraction of the direction removed matches the BF16 intermediate to within 0.00002.
Per-layer measurements are in reencode/reencode-report.json. These are weight-space measurements, separate from the behavioral tests above.
The tokenizer, chat template, vision encoder and built-in MTP layer are NVIDIA's. The model configuration is unchanged apart from a standalone_provenance metadata entry. This is an edited derivative of NVIDIA's quantized checkpoint, not a repeat of its quantization-aware distillation.
Files
| File | Contents |
|---|---|
tensors/, model.safetensors.index.json |
Weights: 21 shards in the base layout |
config.json, tokenizer, chat_template.jinja, preprocessor configs |
Base configuration and processing files, with the provenance addition noted above |
serve-spark.sh |
Single-Spark launcher |
bake-selection.json, frozen-directions.json, frozen-row-norms.json |
Frozen trial 117 intervention inputs |
reencode/ |
MXFP8 re-encode script and per-layer report |
provenance.json |
Build information and validation summary |
SHA256SUMS, release-files.json |
Release file hashes and sizes |
LICENSE, NOTICE, upstream/ |
Governing terms, notices and NVIDIA's original card |
Limitations
- Narrow evaluation: broad capability benchmarks and video input were not evaluated for this derivative. The vision checks were limited to solid colours.
- Profile-specific evidence: single-Spark retrieval checks reached about 30K tokens. The separate two-Spark profile passed the near-262K checks described above. Neither proves reliable performance on every task at those lengths.
- Original targets not met: the project aimed for zero refusals and a KL ceiling of 0.20 and did not meet its original qualification targets. The top-20 KL reported here is distinct from the original full-vocabulary metric.
- Refusals remain: 6 of 100 holdout prompts were detected as refusals.
- Hardware and runtime scope: validation used DGX Spark and pinned B12X runtimes. Other hardware and runtime versions are not qualified here.
- Outputs can be wrong or inconsistent: reduced refusal behavior does not establish correctness, factual reliability or preserved general capability.
Responsible use
Abliteration reduces learned refusal behavior. The model may attempt requests the original declines and can produce harmful, offensive, false or illegal content.
- You are responsible for how you use it and for complying with applicable law and the governing licenses.
- Add your own safeguards before deploying it to end users.
- Do not use it for activities prohibited by the Qwen or NVIDIA terms.
There is no warranty of any kind.
License
This checkpoint is derived from NVIDIA's quantized checkpoint and Qwen's original model. Their governing terms continue to apply; this release does not relicense the base weights.
- NVIDIA Open Model Agreement: the governing download terms for NVIDIA's checkpoint. A copy is included in LICENSE.
- Qwen Community License 1.0: additional terms for the original model, included in upstream/QWEN-LICENSE.
Licensed by NVIDIA Corporation under the NVIDIA Open Model Agreement.
Read the full license texts before use or redistribution, including their commercial-use conditions. Runtime containers and external software have their own licenses.
Citation
@misc{developingdad2026qwen38abliterated,
title = {Qwen3.8-Flash-Next FP4-Local -- Abliterated},
author = {DevelopingDad},
year = {2026},
howpublished = {\url{https://huggingface.co/DevelopingDad/Qwen3.8-Flash-Next-FP4-Local-Abliterated}}
}
@article{arditi2024refusal,
title = {Refusal in Language Models Is Mediated by a Single Direction},
author = {Arditi, Andy and Obeso, Oscar and Syed, Aaquib and Paleka, Daniel and Panickssery, Nina and Gurnee, Wes and Nanda, Neel},
journal = {arXiv preprint arXiv:2406.11717},
year = {2024}
}
@misc{heretic,
author = {Weidmann, Philipp Emanuel},
title = {Heretic: Fully automatic censorship removal for language models},
year = {2025},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/p-e-w/heretic}}
}
Credits
This release stands on other people's work.
Model
- Qwen: Qwen3.8-Flash-Next, the original model, under the Qwen Community License 1.0.
- NVIDIA: the FP4-Local checkpoint, quantized with Model Optimizer (Apache-2.0), under the NVIDIA Open Model Agreement. NVIDIA's published DGX Spark profile is the basis of
serve-spark.sh.
Method
- Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee and Neel Nanda: the refusal-direction research behind abliteration.
- Philipp Emanuel Weidmann (p-e-w) and the Heretic contributors: Heretic (AGPL-3.0). Its search parameterization was reimplemented here, and no Heretic code is included. The Heretic-DGX fork was consulted while adapting it.
- Maxime Labonne (mlabonne): the abliteration write-up and the two prompt sets used to find directions and measure refusals. Neither prompt set is redistributed here.
mlabonne/harmful_behaviors, revision01cead0, comes from AdvBench (llm-attacks, MIT).mlabonne/harmless_alpaca, revision02c6a92, comes from Stanford Alpaca (CC BY-NC 4.0).
- Open Compute Project: the Microscaling (MX) format specification that MXFP8 follows.
Software and serving
- vLLM (Apache-2.0): the serving engine.
- FlashInfer (Apache-2.0) and B12X by Local Inference Lab (Apache-2.0): the kernels.
- eugr/spark-vllm-docker (MIT): the DGX Spark vLLM image used for building and serving.
- Optuna (MIT), PyTorch (BSD-3-Clause) and safetensors (Apache-2.0).
- diffbot: the tuned-stack recipe behind the optional single-Spark speed figure.
- XtraSaltyDev: the separate two-Spark serving recipe, evaluation harnesses and deployment results.
External projects retain their own licenses. This release is not endorsed by Qwen, NVIDIA or any of the projects above.