Why small models (<3B) hallucinate on facts

Short answer

Small models (under 3B parameters) don’t have enough capacity to memorize the world reliably. They confidently make up plot summaries, wrong dates, invented papers, fake quotes. Use them for structure and simple tasks, not for facts.

Detailed explanation

A model’s knowledge is stored in its weights. More parameters = more knowledge that can be packed in. Rough scale of what fits reliably:

• 0.5B: extremely limited. Basic language, no serious knowledge. Only useful for narrow tasks (title generation, tag extraction).
• 1B: light general knowledge. Common facts okay, specifics unreliable.
• 3B: decent general knowledge for common topics. Weak on specialist knowledge, non-English cultures, recent events.
• 7B: broadly knowledgeable. Handles most factual questions with reasonable accuracy in mainstream areas.
• 13B: significantly more depth. Better on niche topics, less confident hallucination.
• 30B+: expert-level breadth. Reliable on most academic domains.

What hallucination looks like:

Ask Qwen 2.5 0.5B “summarize Hans Christian Andersen’s story ‘The Shadow’” - it will produce a confident, detailed summary that is almost entirely fabricated. Plot elements invented, characters wrong, ending made up. The model doesn’t know it’s wrong - this is how it generates when knowledge is thin.

Same question to Llama 3 8B - mostly accurate summary, maybe a small detail off. Same question to 30B - accurate.

Why smaller models sound so confident:

Models generate tokens by predicting the most likely next word. When knowledge is absent, the model generates whatever sounds like it should come next in that kind of text. A summary about a book keeps producing summary-like sentences even if the underlying facts don’t exist. There’s no internal “I don’t know” state to activate.

When small models still shine:

• Formatting tasks (JSON output, markdown structure, list generation)
• Simple language transformation (translation of common phrases, rewriting)
• Extraction (find dates in text, pull out names)
• Fast interactive UX where speed matters more than depth
• Programmatic use (title generation for chats, tagging)

When NOT to use small models:

• Any fact-checking, research, homework
• Anything about specific real-world events, people, works
• Anything requiring specialist knowledge (law, medicine, history)
• Anything you’ll act on without verifying

Why it matters

Users who run their first small model on a laptop often draw wrong conclusions about “local AI is dumb” - the problem is they picked a 1B for tasks that need 8B+. A 7B model can be genuinely useful for research questions. A 1B model is entertainment.

In Abliteration specifically

The Model info tooltip shows a warning line for models under 3B: ⚠ Small model (<3B) — may hallucinate on factual questions.

The warning is smart about MoE - a 30B MoE with ~2B active per token gets a different warning: ⚠ MoE model: 30B total, only ~2B active per token — answers may be shallower than a dense 30B model. Not the small-model warning, because the total 30B does actually store 30B worth of knowledge (even if only some experts fire per token).

Related

• MoE vs dense - the special case for the warning
• How to evaluate for your rig - why not to pick the tiniest model that fits
• Recommended models by RAM - matching model tier to your machine