license: mit
base_model:

• deepseek-ai/DeepSeek-V4.1-Flash-1017
• ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF
  pipeline_tag: text-generation
  library_name: gguf
  tags:
• deepseek
• gguf
• moe
• strix-halo
• amd
• rocmfpx
• expert-pruning
• low-bit

DeepSeek-V4.1-Flash-1017 for Strix Halo (128 GB)

Made for one AMD Strix Halo with 128 GB of unified memory

Part	Size	Where it lives
Model (attention, router, kept experts)	about 97 GiB	GPU memory
Engram lookup table	110.6 GB	stays on the SSD, read on demand (needs no GPU memory)

Only about 97 GiB has to fit in memory. The large size Hugging Face shows on the right side of this page (and its "will it fit" hardware check) adds up every file in the repository: two builds plus the Engram file. That is download size, not memory use. You download one build (about 215 GB on disk), and only about 97 GiB of it is loaded into GPU memory.

Vote on what we quantize next: harshavardhan3002.github.io/ROCmFPX. Big models squeezed to fit Strix Halo like this one, or smaller models done well? One click on GitHub, and it decides what we work on next.

This is a first iteration. It fits the model into 128 GB by removing the least-used experts and reusing existing 1 to 2 bit quantizations, and that costs quality (see the perplexity below). We are actively working on new ROCmFPX formats at around 2.1 bits per weight and below 2 bits that should keep much more of the original quality and run fast on AMD GPUs. This will take time, and we need the community's help: run the files, report speed and quality, and tell us which models you want next.

Status: community test release. It is checked on the CPU and on an NVIDIA GB10 with CUDA (perplexity and GSM8K below), but it has not been run on a Strix Halo yet. Please report what you see (speed, memory, quality) in the Community tab.

What this is

DeepSeek-V4.1-Flash-1017 cut down to fit one AMD Ryzen AI Max+ 395 (Strix Halo) box with 128 GB of unified memory, aiming at a 64K context. The full model has 384 routed experts per layer and needs about 230 GB even at the smallest standard quants. These builds keep the most useful experts in every layer, quantized with IST-DASLab's GSQ-RCO method, so the part that has to sit in GPU memory is about 97 GiB. The large Engram lookup table stays on the SSD and is read on demand.

Files

Each build is 4 files: shards 1 to 3 are the model, shard 4 is the Engram (byte-identical in both builds). Download one folder and point the runtime at shard 1.

Perplexity: 8 chunks of 2048 tokens of mixed English Wikipedia text and source code, CPU, same text for all three rows. The loss against the full model is clear; this is the price of fitting into 128 GB. Keeping more experts at 1.75 bit beat keeping fewer at 2 bit.

Test results

• GSM8K method: llama-server with the embedded chat template, greedy decoding, up to 6144 output tokens, 4 requests in parallel. Each question ends with "Solve the problem step by step. End your reply with a last line of the form 'Final answer: '." and that number is scored. We have no full-model score with the same setup to compare against. GSM8K is easy for a model of this size, so it shows that the pruned builds still reason correctly, not that they match the full model on harder tasks.
• Where it ran: an NVIDIA DGX Spark (GB10, CUDA), because that is where we have the compute. These are not Strix Halo numbers. Generation speed (tg128) was about 12.6 t/s, but other CPU jobs were running on the machine at the same time, so treat it as a lower bound.
• CUDA vs CPU check: same 4 chunks of text on both backends. The two agree on the top token 98.0% (IQ1_M) and 97.2% (IQ2_XXS) of the time, mean KL divergence 0.004 and 0.006, so the results do not depend on the backend.

What is inside

• Routed experts: IST-DASLab's GSQ-RCO quantized experts (ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF), copied unchanged, then cut to the kept experts.
• Which experts are kept: ranked per layer by the REAP score (router weight times expert output size, averaged over the tokens that reach the expert), measured on the full native model over 131,072 tokens of mixed text and code. The kept lists are in keep/. The 320-expert build keeps 97.5% of the routing weight, the 277-expert build 94.4%. 47 experts were never used at all in the calibration run.
• Everything else (attention, router, shared expert, embeddings): Q8_0, quantized from the original checkpoint. The router and its bias are cut to the same experts.
• Engram table: IQ4_NL, 110.6 GB, kept in host memory and read lazily from the file.
• Chat template: embedded (the ISTA-DASLab release's template, adapted from Unsloth's V4.1 release).

How to run

These files need a runtime with DeepSeek-V4.1 support. Upstream llama.cpp does not have it yet; use the ROCmFPX fork, which includes it together with Strix Halo Vulkan tuning:

git clone https://github.com/HarshaVardhan3002/ROCmFPX
cd ROCmFPX
cmake -B build -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j

hf download Harsha001/DeepSeek-V4.1-Flash-1017-GSQ-RCO-StrixHalo-GGUF --include "IQ1_M-E320/*" --local-dir .

./build/bin/llama-server \
  -m IQ1_M-E320/DeepSeek-V4.1-Flash-1017-StrixHalo-IQ1_M-E320-00001-of-00004.gguf \
  -ngl 99 -lm mmap -c 65536 --jinja

Notes:

• Keep -lm mmap (also for llama-bench and llama-cli). Without it the runtime turns memory mapping off on integrated GPUs, and Strix Halo counts as one. It then reads the whole 110 GB Engram table into RAM, which does not fit next to the model. With -lm mmap the table stays in the file and only the rows each token needs are read.
• The GPU must be able to use about 100 GiB. On Windows set the largest graphics memory in the BIOS or AMD Software that still leaves the system enough RAM; on Linux raise the GTT limit. Close other GPU programs.
• The Engram table is always placed in host memory by the runtime, even with -ngl 99. Keep the files on a fast NVMe drive.
• Quick speed test: ./build/bin/llama-bench -m IQ1_M-E320/DeepSeek-V4.1-Flash-1017-StrixHalo-IQ1_M-E320-00001-of-00004.gguf -ngl 99 -lm mmap -p 512 -n 128
• Expect low speed at first: the 1 to 2 bit expert kernels on Vulkan are not tuned yet. Faster kernels for these types are the next step in the ROCmFPX fork.

Credits

• Original model: deepseek-ai/DeepSeek-V4.1-Flash-1017, MIT license.
• Expert quantization: IST-DASLab, GSQ (Gumbel-Softmax Quantization, arXiv:2604.18556) and RCO (Riemannian Constrained Optimization, arXiv:2605.00649). Their release is MIT licensed.
• DeepSeek-V4.1 runtime support: llama.cpp pull request #30817 (Daniel Han), included in the ROCmFPX fork.
• Expert ranking: REAP criterion (router-weighted expert activation pruning).

This repository only selects and repackages those weights; all credit for the model and the quantization method goes to their authors.