MoE vs dense models

Short answer

Dense models use all their parameters for every token. MoE (Mixture of Experts) models only activate a small subset per token. That means MoE runs at the speed of the active parameters (fast) but takes RAM for the total parameters (heavy). Qwen3-30B-A3B is 30B on disk, ~3B in speed.

Detailed explanation

Dense model: every one of its parameters participates in generating every token. A 7B dense model does ~7B computations per token. Simple, predictable - bigger model = slower per token, better answers.

MoE (Mixture of Experts) model: the layers include multiple “experts” - usually 8, 16, or more - but only a few (typically 2-8) are active for any given token. The router picks which experts to use per token. A 30B MoE with 8 experts and 2 active per token computes only ~3B parameters worth of math per token, even though all 30B are loaded into RAM.

The trade-off:

Property	Dense 30B	MoE 30B (A3B, 2 of 8 active)
File size	~20 GB (Q4)	~20 GB (Q4)
RAM to load	~20 GB	~20 GB (all experts resident)
Speed (tok/s)	~2 tok/s	~60 tok/s
Quality on hard tasks	High (30B-level)	Lower - closer to a good 7B
Quality on easy tasks	High	Also high

Why MoE isn’t a free win: the 30B parameters are there, but any given token only sees ~3B of them. On reasoning-heavy questions where the model needs deep, coherent thinking, a dense 30B can outperform an MoE 30B - the dense model gets its full brain on every token. On broad-knowledge questions where the router picks the right expert, MoE matches or beats dense of similar active size.

Popular MoE models:

• Qwen3-30B-A3B - 30B total, ~2B active. Runs like a 3B in speed, knows like a 15B.
• Mixtral 8x7B - 47B total, ~13B active. Runs like a 13B, knows like a 30B.
• DeepSeek-V2 lite - similar pattern.

Why it matters

Two identical-sized files on disk can perform completely differently. A dense 30B on your laptop is a novelty (2 tok/s = painful). A MoE 30B is a real tool (60 tok/s = comfortable). But knowing you have enough RAM for a 30B file is only half the equation - check whether it’s MoE first.

In Abliteration specifically

Abliteration detects MoE from the GGUF header (expert_count field) and shows the split in the Model info tooltip: Qwen3-30B-A3B: 30B total, only ~2B active per token. Expected tok/s uses the active parameter count, so you see a realistic ~60 tok/s instead of a misleading dense-30B estimate.

The fit-check dialog uses the file size (all experts must be in RAM) so it correctly warns you that a 30B MoE needs ~20 GB even though it runs like a small model.

The “small model may hallucinate” warning is smart about MoE - it doesn’t fire for a 30B MoE just because the active params are ~2B. Instead it shows a specific MoE note: “30B total, only ~2B active per token - answers may be shallower than a dense 30B model. Speed matches ~2B though.”

Related

• How to evaluate for your rig - MoE breaks simple “file size = performance” assumptions
• Expected tok/s - active params drive this number
• Small model hallucinations - related but distinct warning