Understanding your rig
Local LLMs are bottlenecked by RAM first, CPU second.
Short answer
Local LLMs are bottlenecked by RAM first, CPU second. Your free RAM (not total) decides which models load smoothly today. GPU matters on Apple Silicon Macs, where Metal handles the heavy work automatically; everywhere else, it is CPU that carries the load. The same rules and the same “Your rig” popup apply on Windows and macOS.
Detailed explanation
What each hardware number means for LLMs:
- RAM total. How much physical memory your machine has. Sets the absolute ceiling, you cannot run a 40 GB model on a 32 GB machine, period.
- RAM free. What is not currently used by the OS, browser, or open apps. Sets the practical ceiling right now, a 20 GB model needs at least 20 GB free.
- CPU name. Modern CPUs (Intel Core Ultra 7 or 9, AMD Ryzen 7 or 9 7000-series, Apple M-series) generate at 15 to 30 tokens per second on 7B models on Windows and Intel Macs, and thirty or more on Apple Silicon with Metal. Older CPUs (pre-2020, dual-core, no AVX2) run 3 to 10 times slower.
- CPU cores or threads. More threads means faster generation up to about 8 to 12 threads. Beyond that, diminishing returns.
- GPU. Named on the Hardware screen whatever it is. On Apple Silicon Macs, Metal is on by default and the M-series GPU cores do most of the work. On Windows and Intel Macs, only modern NVIDIA cards accelerate inference today; everything else runs on CPU with a
GPU acceleration is not available on this PCline under the model tiers. See Hardware detection for the specifics, Apple Silicon acceleration for what Metal does, and CUDA and Vulkan backends for the broader Windows story. - Disk free. Where downloaded models live. Each GGUF is 1 to 20 GB, they add up.
How rig shapes what you can run:
| Total RAM | Realistic ceiling | Sweet spot |
|---|---|---|
| 8 GB | 3B Q4 | 3B Q4 |
| 16 GB | 7B Q4, 13B Q3 | 7B Q4 |
| 32 GB | 30B MoE, 13B Q4 | 13B Q4 or 30B MoE |
| 64 GB | 70B Q4 | 30B dense or 70B Q4 |
Realistic ceiling = biggest thing that will load. Sweet spot = biggest thing that runs fast enough to be pleasant.
Why free RAM matters more than total:
A 32 GB laptop with Chrome, Slack, VS Code, and Spotify open easily has 20 GB used. That leaves 12 GB free. So realistically that laptop is a 7B Q4 machine right now, not a 30B machine. Total RAM tells you what is possible, free RAM tells you what will work today.
CPU speed rough guide (7B Q4):
- Modern flagship laptop (Core Ultra 9, Ryzen 9 8945HS): 20 to 30 tok/s
- Modern mainstream (Core Ultra 7, Ryzen 7 8845HS): 15 to 25 tok/s
- Apple Silicon (M1 or newer, Metal on): 25 to 45 tok/s
- Older laptop (i5 or i7 10th to 11th gen): 8 to 15 tok/s
- Intel Mac, Core i5 or i7 2019 to 2020: 3 to 10 tok/s
- Old or low-end (pre-2020 i3, older): 3 to 7 tok/s
Why it matters
Understanding rig limits saves time. You’ll know to skip 30B models on your 16 GB laptop rather than downloading 20 GB just to find out. You’ll know that closing Chrome before loading a big model actually matters.
In Abliteration specifically
The Hardware card (Settings > Hardware) shows all these numbers in one place, plus a “You can run” line that translates them into model-tier recommendations for your specific machine. Free RAM and free disk refresh every time you open that screen; the other numbers are cached from launch. Nothing is uploaded, everything is read from local OS APIs.
On model add, the fit-check dialog does the math per model, so even without memorizing rules the app tells you upfront whether a specific download will work.
Related
- Hardware card - the Settings screen with all this info
- RAM requirements - the formula behind “will it fit”
- CPU vs GPU - why non-NVIDIA GPUs are informational on Windows
- Apple Silicon acceleration - the one platform where GPU already matters
- Free vs total RAM - the two numbers explained