CPU vs GPU inference (why Abliteration is CPU-only right now)

Short answer

Abliteration runs models on your CPU only right now. A GPU would be 5-20× faster, but CPU-only means no NVIDIA/AMD-specific setup - the app works the same on any Windows machine. GPU support is planned.

Detailed explanation

Language models can run on CPU or GPU. GPUs are much faster because generation is heavily parallelizable - each token needs thousands of matrix multiplications, and GPUs do those in bulk. A 7B model that runs at 15 tok/s on a modern CPU can hit 100+ tok/s on a mid-range NVIDIA GPU.

The trade-off is compatibility. GPU inference needs either CUDA (NVIDIA only), Vulkan (broader - NVIDIA + AMD + Intel Arc), or Metal (Apple only). Each backend has its own runtime dependencies, driver requirements, and quirks. Shipping a Windows app that “just works” on any hardware is much easier CPU-only.

Abliteration’s approach: start with CPU-only for universal compatibility, add GPU as a separate downloadable backend later. The user chooses to upgrade - they’re not forced to install CUDA before the app runs.

Practical CPU speed depends on:

• Cores/threads - more is faster, but with diminishing returns after 8-12 threads
• RAM bandwidth - LPDDR5x > DDR5 > DDR4
• Model size - 3B fits in cache better than 30B
• Quantization - Q4 is faster than Q8 (less data to move through cache)

Rough tok/s on modern laptop CPUs (Intel Core Ultra 7, AMD Ryzen 7 8xxx, Apple M-series when running Windows on ARM):

• 3B Q4: 40-80 tok/s (feels instant)
• 7B Q4: 12-25 tok/s (feels natural)
• 13B Q4: 5-10 tok/s (noticeable wait)
• 30B Q4 dense: 1-3 tok/s (slow; use MoE variants instead)

Why it matters

If another local runtime feels faster on the same machine, it is almost certainly using the GPU. Abliteration runs on CPU today on Windows and Intel Macs; on Apple Silicon it uses Metal by default and the difference disappears. If you have a modern NVIDIA card sitting idle on a Windows machine, Abliteration is not using it yet - CUDA is on the GPU roadmap.

If you don’t have a discrete GPU (integrated Intel Arc, or thin laptop with no GPU at all), you’re not missing anything - Abliteration uses what everyone has.

In Abliteration specifically

The Hardware card names whatever GPU your machine actually has - Intel Arc Graphics, AMD Radeon RX 6600, NVIDIA GeForce RTX 4070, whatever Windows reports. When the app can accelerate on it (modern NVIDIA only, driver branch >= 450), the name shows in green with VRAM. Everything else is named in amber, and a one-line disclaimer below reads GPU acceleration is not available on this PC. Abliteration runs on your CPU. See Hardware detection for the specifics.

Expected tok/s in the Model info tooltip is calculated for CPU inference on your specific hardware. Note: the tok/s tooltip label may name your NVIDIA GPU in its “on your {hardware}” phrasing even though the estimate itself is a CPU one - the estimator is CPU-only across the board.

Related

• Expected tok/s - the number you actually see
• GPU roadmap - what’s coming and when
• Slow generation - when CPU-only is the bottleneck