Blind Spots of Frontier Models: Nanbeige4-3B-Base
This dataset was developed for the Fatima Fellowship 2026 Technical Challenge. It explores the natural vulnerabilities and structural blind spots of a raw, unaligned 3B parameter base model.
1. Model & Environment
- Model Tested: Nanbeige/Nanbeige4-3B-Base (Released Feb 2026)
- Loading Method: Google Colab (T4 GPU),
bfloat16precision,transformerslibrary. Greedy decoding (do_sample=False) was used to isolate deterministic base-model behaviors without sampling randomness.
2. Analysis of Blind Spots
Testing 15 diverse logic constraints revealed that while the model possesses high baseline intelligence (zero-shot exponential math, temporal logic), it suffers from severe structural blind spots:
- Format Hijacking & Persona Bleed: The model frequently abandons the user's prompt to adopt personas from its pre-training data. It turned a gender-bias prompt into a multiple-choice English exam, and turned the end of a story into a literary critique.
- Tokenization Failures: The model fails at character-level manipulation (spelling "MODEL" backwards), exposing that its tokenizer forces it to view words as opaque chunks rather than individual letters.
- Chat Template Ignorance: When fed explicit
<|im_start|>and<|im_end|>tokens, it answers correctly but fails to halt generation, instead hallucinating the user repeating the question. - Pre-training Data Leaks: Surprisingly, the model frequently leaks
<think>tags mid-generation to fact-check its own hallucinations or solve math. This strongly implies the base pre-training corpus was heavily saturated with Chain-of-Thought reasoning traces. - Unaligned Safety & Missing Guardrails: Lacking Reinforcement Learning from Human Feedback (RLHF), the model possesses no safety filters or refusal mechanisms. When prompted with an illicit task (e.g., how to hotwire a car), it cheerfully provided step-by-step instructions rather than declining the request.
3. Proposed Fine-Tuning Strategy
To transform this base model into a reliable assistant and fix these specific errors, a two-stage alignment pipeline is required:
A. Supervised Fine-Tuning (SFT) for Format Adherence:
The model must learn to respect chat templates (like <|im_start|>) and output an End-Of-Sequence (EOS) token the moment a question is answered. This will cure the "Format Hijacking" and "Stop Token Failures".
B. Direct Preference Optimization (DPO) / RLHF for Behavioral Control:
To fix the safety vulnerabilities (hotwiring instructions) and suppress the raw <think> tag leaks, preference tuning is needed. DPO will teach the model to assign higher probability to concise, aligned answers and penalize degenerative loops and unprompted internal monologues.
4. Dataset Assembly & Scale
- How to Assemble: Because manual annotation is expensive, I would use Synthetic Data Generation. I would write system prompts for a larger, aligned frontier model (e.g., GPT-4o or DeepSeek-V3) to generate thousands of high-quality conversational turns. To fix the tokenization blind spot, I would explicitly synthesize character-level spelling and word-manipulation tasks.
- Required Size: For a 3B parameter model, I estimate an SFT dataset of 20,000 to 30,000 highly diverse conversational examples is needed to instill proper chat formatting and stop-token behavior. Following that, a DPO dataset of 5,000 to 10,000 chosen/rejected pairs would be required to polish the safety guardrails and eliminate formatting loops.