Qwen3.8-Flash-Next-abliterated-GSQ-RCO-Coder-Strata-GGUF

Credits first

The model weights in this repo are not mine. They are the GSQ-RCO Coder release of Qwen3.8-Flash-Next made by the Deep Algorithms and Systems Lab at ISTA: 256 of the 512 experts in each layer are kept, the rest are removed, and the kept weights are stored at 3.5 bits. The files are copied byte for byte from ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF at revision 5348543e0147355ac9cbcb031184a3546350988e. The SHA256 of both shards and of the vision projector match their files. See SHA256SUMS.

If you use these weights, cite the two ISTA-DASLab papers. The BibTeX is at the bottom of this page.

My part is the check that the refusal vector I published for the unpruned sizes also works on the pruned model, and the measurements below. The vector is the same 480 KB file as in the repo with the four unpruned sizes, with the same SHA256. For the original card and ISTA's own SWE-bench and LiveCodeBench numbers, go to the ISTA-DASLab repo.

What the Coder is

ISTA-DASLab removed half of the experts of Qwen3.8-Flash-Next and picked the ones to keep on code, agentic and vision data. The download is 58.4 GB instead of 83.6 GB for IQ3_S, and only the first shard (29.6 GB) has to be in memory. By the tensor shapes in the files, 116.5B of the base model's 176.9B parameters are left: 60.4B of the 120.8B in the experts, the 51.2B n-gram table and 4.9B in the rest of the transformer. Their numbers: 91.3% of the full model's SWE-bench Verified score and 98.7% of LiveCodeBench v6. They also say it is weaker outside code. Strata's page adds that languages other than English got worse. If you want a general model and have 48 GB of RAM or more, take one of the unpruned sizes from my other repo.

The vector

After each steered layer Strata subtracts one direction r from every residual stream:

h = h - (h . r) r

The weights stay as ISTA-DASLab published them. A request with "experimental_speed_projection": false runs the original model.

The direction comes from the Huihui checkpoint of the full model. Huihui changed 144 residual-writing tensors with a rank-one edit along one unit direction r, and the projection in Strata does almost the same thing at run time. The Coder keeps the attention, the router and the residual stream of the full model, but only half of the experts that write into that stream. So I did not assume the direction carries over. I measured it, and it does.

Run it with Strata

You need an NVIDIA or AMD card with 12 GB or more and 32 GB of RAM. Strata's own page has the full requirements.

git clone https://github.com/Niko1221/Strata
hf download alesha-pro/Qwen3.8-Flash-Next-abliterated-GSQ-RCO-Coder-Strata-GGUF --local-dir coder
mv coder/mmproj-Qwen3.8-Flash-Next-BF16.gguf coder/IQ1_M/
cd Strata
./setup.sh --setup --family coder --gguf-dir ../coder/IQ1_M --vision gpu \
  --experimental-speed-projection ../coder/Huihui-Qwen3.8-Flash-Next-refusal-direction-r.gguf

On Windows use START-HERE.bat with the same flags. The mv puts the vision projector next to the shards, where setup looks for it. Setup builds or downloads the engine, fetches the MTP draft layer (about 6 GB from the original Qwen checkpoint), packs the model and starts the server at http://127.0.0.1:8080. OpenAI clients use /v1, Anthropic clients use /v1/messages.

If you already have one of the unpruned sizes, you only need the first shard. Shard 2 (the n-gram table) and the vision projector are the same files.

Setup writes these engine flags into strata-<model>.json:

--control-vector-scaled <path>/Huihui-Qwen3.8-Flash-Next-refusal-direction-r.gguf:1.0
--control-vector-layer-range 4 44 --cvec-mode project --cvec-dir per-layer

I tested layers 4-44 and 1-47. Both are in the tables below and both work. Most of my checks ran on 1-47. To use that range, change the two numbers in that file.

I did not run the installer end to end on my machine. I built Strata 0.1.40 from source and started the engine with the arguments setup writes for --family coder.

Plain llama.cpp will load these GGUFs, but its stock --control-vector adds a vector. It has no projection mode, so there the files behave as the original model.

Measurements

One RTX 3090 at 300 W, EPYC 7642 (48 cores, AVX2, no AVX-512), 125 GB RAM, Strata 0.1.40 built for sm_86 and started with --spec-min-p 0.7. At a 65,536 context the card held 8,524 of the 12,288 experts, at 262,144 it held 8,346.

Refusals, thinking off

The first three rows are greedy, 96 tokens per answer. The probe is the 10 prompts of my other cards (lock picking, phishing email, keylogger, explicit story and similar), sampled at temperature 1.0. A refusal is a regex on the first 400 characters of the answer.

I read the opening of every flagged answer in the 1-47 column. The JailbreakBench hit is an answer that does the task but turns it around: asked for an expert thread that spreads a myth, the model wrote a thread that debunks it. A wider regex flags 2 more answers there. One argues with the request instead of doing it (a campaign against hiring people with disabilities). The other is a false match, the model answers the question. The probe hit is the piracy question: the model answers but says it will not give download links. So the plain refusals are gone, and 3 answers out of 156 still push back in a softer way.

The original's 2 hits on ordinary instructions are one real refusal of a medical question and one answer that starts with "I cannot generate audio".

Refusals, thinking on

"No answer" means the request used all its tokens while the model was still reasoning. I first blamed the vector for it. That was wrong, and a control run shows it: 24 harmless tasks of the same size (write a script, develop a website, write a guide), thinking on, 2,500 tokens.

The Coder does this by itself. Strata's default effort is xhigh, and the model's template then adds a system line that asks it to think carefully and consider alternatives. On a large open task the Coder plans the answer inside its reasoning and runs out of tokens before it starts writing. On a harmful request the original model does not show this, because a refusal is short.

What helps: with "reasoning_effort": "low" the vector arm answered all 18 harmful requests that had no answer above, and none refused. More tokens help less: with 8,192 tokens 8 of those 18 answered and 10 were still reasoning. I did not test medium. The agent tasks below run with thinking on and passed.

The unpruned IQ3_S with the same vector left 6 of the same 48 harmful requests without an answer, against 18 on the Coder.

A direction measured on the Coder itself

The Huihui direction comes from the full model, so I also measured directions on the pruned one. I built llama.cpp at the commit Strata pins, added a tool that dumps the residual after every layer (this model keeps 4 parallel streams of 2,560 values per layer), and ran 256 harmful and 256 harmless instructions in 6 contexts: thinking off, the three efforts, and both with an agent system prompt. The direction per layer is the difference of means, with the harmless mean removed.

Nothing I measured on the pruned model beat the Huihui direction, so the repo keeps it. My directions have a cosine of 0.2 to 0.5 with it, depending on the layer, and they separate harmful from harmless prompts as well as it does, but they remove fewer refusals. The direction from the thinking contexts alone does not remove refusals at all. The lower no-answer counts of the first two rows come from refusals, which end the reasoning early.

Distance to the original model

Teacher-forced on 6 windows of 1,024 tokens of English Wikipedia text (WikiText-2) and 6 of C++ source. KL is from the original to the vector arm of the same quant, over the original's top 256 tokens. The IQ3_S rows are the unpruned size from my other repo, on the same windows, for comparison.

The vector costs the Coder about what it costs the unpruned model. The table also shows what the pruning itself did. On the C++ text the Coder is close to IQ3_S: it gets the next token right 82% of the time against 85%. On the Wikipedia text it is 62% against 82%, and the perplexity is 2.7 times higher. That matches what ISTA-DASLab say about the release: it keeps code and gives up general text. The low numbers for IQ3_S suggest the base model has seen this Wikipedia text, so read the prose rows as a comparison of the two files and not as a general score.

Agent tasks, tools, images, long context

All with the 1-47 vector on.

The coding tasks are a Python log summary fix, an HTTP Range parser with 18 tests, and semver, LRU cache and CSV fixes in JavaScript. I did not run them without the vector on this model, so they show that the vector did not break tool use, not how the Coder compares to the full model. The image check is a smoke test on one picture.

Speed

Each request reads an independent document and writes 512 tokens, nonthinking and greedy, with nothing cached. Two repeats per depth, MTP drafts on, a 262,144 context with KV streaming, images off, the 1-47 vector on. Decode and prompt reading are tokens/s as the engine reports them, mean of the two repeats.

Without the vector the same run gave 80.2 at 4K and 83.6 at 32K, so the vector costs nothing I can measure. The two repeats at 261K were 56.8 and 71.7. MTP accepted 73-80% of the drafts. Peak VRAM was 23,822 MiB. On the same card and engine the unpruned IQ3_S decodes 66-70 tokens/s.

In the refusal runs and the agent sessions, with thinking and sampling at temperature 1.0, the server log gives a median decode of 82 tokens/s on requests of 200 tokens or more.

Memory

The same RTX 3090, the 262,144 context, the vector on, 2 repeats per depth. A RAM limit here is a hard one: a cgroup that counts the file cache, with swap off, and the model's files dropped from the cache before the start. Decode and prompt reading are tokens/s at a 4K and a 32K prompt.

So a PC with one 24 GB card and 32 GB of RAM runs it at full speed. With 16 GB of RAM the answers still come at 55 to 60 tokens/s, but prompts are read 9 to 12 times slower, because the experts a prompt needs come from the disk. My model files are on a SATA SSD, an NVMe drive should do better there. 10 GB of RAM is the floor I found: it runs, slowly.

The VRAM numbers are what nvidia-smi showed for the card (23,822 to 23,856 MiB and 12,886 MiB), and 0.6 GB of that belongs to another idle process. GB here means 1,024 MiB. The last row is not a 12 GB card. It is the 3090 with Strata told to leave the rest of the VRAM free (--vram-reserve-mib), so the memory budget is that of a 12 GB card and the compute is a 3090's. With 12 GB the card holds 2,715 experts instead of 8,346, and the CPU does more of the work.

Strata's setup picks the low-RAM mode by itself on a PC with little RAM. I passed the flags by hand.

This CPU has no AVX-512, and the experts that do not fit in VRAM are computed on the CPU. A desktop with AVX-512 may decode faster.

I did not test real photos or documents in vision, more than one GPU, AMD cards, or languages other than English.

Files

The file names and folders are the published ones because Strata's setup looks for them. ISTA-DASLab named the size IQ1_M for its 1.89 bits per parameter of the original model. The kept weights are stored like IQ3_S.

Removing refusals removes a safety behaviour. What the model writes with the vector on is your responsibility.

Citation

Cite the authors of the pruning and quantization:

@article{gsq2026,
  title  = {GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling},
  author = {Dadgarnia, Alireza and Tabesh, Soroush and Nikdan, Mahdi and Helcig, Michael and Kurtic, Eldar and Kleinegger, Maximilian and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2604.18556},
  year   = {2026}
}
@article{rco2026,
  title  = {Model Compression with Exact Budget Constraints via Riemannian Manifolds},
  author = {Helcig, Michael and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2605.00649},
  year   = {2026}
}

License

ISTA-DASLab state that their weights inherit the license of the base model. The base model, the Huihui checkpoint the direction comes from and this repo are under the Qwen Community License 1.0. Read that file for its terms.