How a token becomes an answer (inference pipeline)

Short answer

Your message goes through five stages before you see a reply: tokenization, prompt formatting, prefill, generation, detokenization. The pulsing cat represents generation. The wait before it appears is prefill.

Detailed explanation

1. Tokenization. Your message (“How are you?”) is split into small chunks called tokens (“How”, ” are”, ” you”, ”?”). Every model has its own tokenizer baked into the GGUF file. English averages ~0.75 words per token; Russian, Chinese, code all have different ratios.

2. Prompt formatting. The tokenized message is wrapped in the model’s chat template. Llama 3 uses <|start_header_id|>user<|end_header_id|>...<|eot_id|>. Qwen uses <|im_start|>user\n...<|im_end|>. Getting this wrong is the #1 cause of “the model produces gibberish” - Abliteration auto-detects the template from the GGUF.

3. Prefill. The model reads the entire formatted prompt at once (including all previous messages in the chat, plus system prompt) and builds up its KV cache. This is fast per token but there are many tokens, so on CPU it takes 0.5-5 seconds for a fresh chat and can be 10+ seconds if the chat is long. During prefill, you see the “Loading…” or “Thinking…” indicator with no visible output.

4. Generation. The model produces tokens one at a time, each token conditioned on everything before it. This is the phase where tokens per second (tok/s) is measured. The pulsing cat + streaming text you see is generation happening live.

5. Detokenization. As tokens stream in, they’re converted back into human-readable text and displayed. This is instant - no visible cost.

Reasoning models (DeepSeek R1, Qwen QwQ) add a hidden phase: they generate <think>...</think> tokens first (visible in Abliteration as the collapsible R block), then the final answer. From the pipeline’s perspective this is all just generation - it’s the model choosing to spend tokens on internal reasoning before answering.

Why it matters

Understanding where time goes helps you diagnose speed issues. Long “loading…” with no output = prefill on a long chat (or slow disk swap). Slow token streaming after that = generation-bound (CPU speed, model size). Weird symbols in output = tokenization/template mismatch. Sudden freeze mid-generation = usually RAM exhaustion (KV cache hit a wall).

In Abliteration specifically

The generating state shows two indicators simultaneously: a “Loading {Model Name}…” or “Generating…” text label, and a small pulsing cat animation. Both appear together - not one or the other.

Cancel (Stop button) behavior differs by stage: during generation (stage 4), pressing Stop halts cleanly and keeps whatever was already streamed. During model load or prefill (stages 1-3), the cancel request is not honored - Abliteration waits for prefill to complete before it can honor the stop. This can mean a several-second delay on a long chat between clicking Stop and generation actually halting.

The tok/s shown in the model info tooltip is an estimate for stage 4 only. It ignores prefill time, which is why a 45 tok/s model can still feel slow if you’re pasting a huge chat history - you’re paying a big prefill cost before the 45 tok/s clock starts.

Related

• KV cache - built during prefill, grows during generation
• Expected tok/s - measures generation phase only
• Chat template auto-detection - stage 2 of this pipeline