Generation is very slow

Short answer

If you’re getting under 3 tok/s on a model that should be faster: probably swap-to-disk from too little free RAM. Close some apps and try again. If that doesn’t help, the model may just be too big for your CPU.

Detailed explanation

Step 1 - What “slow” should mean for this model.

Check the expected tok/s in the Model info tooltip (ⓘ icon on the model). That’s what the model should generate at on your CPU. If you’re getting much less than that (e.g. expected 15, actual 2), something is wrong. If you’re getting the expected number and it feels slow, the model is genuinely too heavy for your rig - pick a smaller one.

Step 2 - Check for swap-to-disk.

Open Task Manager → Performance → Memory. If “In use” is close to or above your total physical RAM, Windows is swapping to disk. Every token generation is now reading gigabytes from SSD/HDD. This is 10-100× slower than in-RAM inference.

Fix: close other apps, restart the app, load the model again. If the model + Windows still doesn’t fit total RAM, this model isn’t for this machine.

Step 3 - Check the model type.

Dense 30B model on 32 GB RAM: often 1-2 tok/s even without swap. That’s just how big the model is.

Fix: switch to an MoE 30B (Qwen3-30B-A3B) - same memory footprint but ~10× faster because only ~2B params are active per token. Or drop to a smaller dense model (13B or 7B).

Step 4 - Context length.

Very long contexts (32K, 128K) slow generation slightly - attention has more history to compute over. Not usually the primary cause, but combined with heavy models it compounds.

Fix: if you set the context slider high manually and don’t need it, drop back to Auto.

Step 5 - Long prefill.

If the pause before text starts streaming is long (10+ seconds), that’s prefill on a long chat, not slow generation. The tok/s clock starts after prefill.

Fix: nothing you can do besides shorter chats, or wait for GPU support (prefill is much faster on GPU).

Why it matters

“Slow” has several causes with different fixes. Swap is fixable (close apps). Wrong model size for the rig isn’t (change model). Prefill on long chats isn’t (accept the pause). Diagnosing which one you have determines whether it’s a 30-second fix or a “download a different model” situation.

In Abliteration specifically

The expected tok/s in the ⓘ tooltip is calibrated for CPU inference on your specific hardware. If you’re consistently far below it, RAM swap is by far the most likely cause. GPU support is planned but not shipped - for now, everything is CPU.

MoE models honestly report their fast expected speed based on active parameters. Dense models of equivalent file size will be slower.

Related

• Expected tok/s - the number you’re comparing against
• MoE vs dense - often the answer for “30B is too slow”
• Free vs total RAM - swap-to-disk explained
• First load is slow - related but different problem