GPU support - CUDA and Vulkan (coming)

Short answer

GPU inference isn’t in Abliteration yet. When it lands, you’ll be able to pick between CUDA (NVIDIA only) and Vulkan (NVIDIA + AMD + Intel Arc) as an optional downloadable backend. CPU stays as the default fallback.

Detailed explanation

The plan for GPU support:

CUDA backend - best performance on NVIDIA cards. Cleanest driver story, most mature, but locks you to one vendor. Recommended for anyone with a modern NVIDIA GPU (RTX 20-series and newer).

Vulkan backend - cross-vendor. Works on NVIDIA, AMD Radeon, and Intel Arc GPUs. Slightly slower than CUDA on NVIDIA hardware but only path for AMD and Intel Arc users. Recommended for anyone without NVIDIA.

CPU backend - always available, always the default. Even after GPU support arrives, CPU will remain the fallback for machines without a suitable GPU or when the GPU is busy with something else.

The user picks a backend once in Settings. Downloading it is separate from the main app (GPU backends are large - ~1-2 GB each) so people who don’t want them aren’t forced to.

Speed expectations once GPU backends land:

• CPU-only 7B Q4 at 15 tok/s → NVIDIA RTX 4060 with CUDA → ~120 tok/s (8× faster)
• CPU-only 13B Q4 at 7 tok/s → same GPU → ~50 tok/s (7× faster)
• CPU-only 30B Q4 at 2 tok/s → same GPU → ~15 tok/s (7× faster, becomes usable)

Why it matters

If you’re evaluating Abliteration for eventual production use and have an NVIDIA/AMD GPU, know that the current CPU-only limit is temporary - the architecture is designed to add GPU as a swappable backend.

In Abliteration specifically

Right now: no GPU-related UI beyond the Hardware card, which names whatever GPU is in the machine and, for anything the app cannot accelerate on, shows a one-line disclaimer under the model tiers - GPU acceleration is not available on this PC. Abliteration runs on your CPU. No settings to toggle. No backend picker. See Hardware detection for what qualifies as “usable” today.

When GPU support ships: Settings → Backend, with options CPU / CUDA / Vulkan, download links for each, and per-model preference (some models are worth GPU RAM, others fine on CPU).

Related

• CPU vs GPU - why the split matters and what CPU can do today
• Expected tok/s - the number that jumps 5-20× with GPU