Expected tok/s - what the number means
Tokens per second is how fast the model generates output. 1 token ≈ 0.75 English words.
Short answer
Tokens per second is how fast the model generates output. 1 token ≈ 0.75 English words. So 15 tok/s ≈ 11 words per second - about the pace of natural typing. Above 25 tok/s feels instant; below 5 tok/s feels slow.
Detailed explanation
When you send a message and see text streaming in, each token is one small chunk being generated. Tok/s is the rate that stream is coming out. It measures only the generation phase - not the pause before generation starts (that’s prefill, a separate thing).
How to read a tok/s number:
| tok/s | words/sec | Feel |
|---|---|---|
| 60+ | 45+ | Instant - text appears faster than you read |
| 25-60 | 19-45 | Fast - comfortable streaming |
| 10-25 | 7-19 | Natural - like a typing person |
| 5-10 | 4-7 | Noticeable - you’re waiting a bit |
| 1-5 | 1-4 | Slow - annoying for anything long |
| <1 | <1 | Painful - usually means swap-to-disk |
What controls tok/s:
- Model size - 3B >> 7B >> 13B >> 30B (dense). A 3B might run 4× faster than 13B on the same CPU.
- Quantization - Q4 is faster than Q8 (less data to move through cache).
- CPU cores/threads and RAM speed - modern laptops with LPDDR5x are noticeably faster than older DDR4 systems.
- Context length - huge chats slow generation slightly (more KV cache to attend over).
- MoE vs dense - MoE models generate at the speed of their active parameters, not total. Qwen3-30B-A3B runs like a 3B model (~60 tok/s) despite being labeled 30B.
Why it matters
Tok/s is the single number that tells you if a model is worth using on your hardware. A brilliant 70B model at 0.5 tok/s is unusable for interactive chat. An okay 3B at 60 tok/s is often more useful in practice.
In Abliteration specifically
The Model info tooltip (the (i) icon next to a model) shows tok/s as two lines:
Expected speed: ~45 tok/s (~34 words/sec)on your {hardware phrase}
The estimate is memory-bandwidth based (not raw CPU throughput). For a Q4 model, the estimator computes bandwidth_GB_per_s ÷ file_size_GB - because generation is bottlenecked by streaming weights from RAM, not by arithmetic. This is why modern laptops with LPDDR5x are noticeably faster than older DDR4 systems even at the same core count.
The estimator uses:
- Active parameter count (dense = total, MoE = active experts)
- Detected RAM type and bandwidth
- Q4 as the baseline quant assumption
The number doesn’t include prefill time. On a long chat, the first token can take several seconds to appear even if tok/s afterward is fast.
MoE models like Qwen3-30B-A3B honestly show their ~2B-equivalent speed (~60 tok/s), not the misleading 30B speed. Same file size on disk, very different generation speed.
Note: the “on your {hardware}” phrase may name an NVIDIA GPU if one was detected - but the number itself is always a CPU estimate; Abliteration doesn’t do GPU inference yet.
Related
- CPU vs GPU - the biggest lever on tok/s once GPU support arrives
- MoE vs dense - why a 30B MoE feels like a 3B
- Slow generation - what to try when tok/s is much lower than expected