language:

• sw
• en
• rw
• am
• pt
• rn
• lg
• fr
  license: cc-by-sa-4.0
  task_categories:
• text-classification
• feature-extraction
  tags:
• decision-model
• guardrails
• swahili
• east-africa
• routing
• multi-task
• tanzania
• civic-tech
  size_categories:
• 10K<n<100K

Mwamuzi-Data v0.2: multi-task Swahili-first decision data

Training and evaluation data for a general System 1 decision model, built Swahili-first with multilingual coverage. All rows are in Laya's {state, questions, expected} format. Every question has English and Swahili instruction variants, which share the same label keys.

Tasks

¹ Small authored sets appear 4× in training: in both instruction languages, and each repeated twice.
² Each eval row appears twice, once with English instructions and once with Swahili instructions (tag sw_instructions).
³ eval_langs counts language tags, including source-specific codes (e.g. AfriSenti's sw and SIB's swh_Latn).

Files

train_mix.jsonl            29,350 rows, all commercial-safe tasks, shuffled (use this to train)
train/<task>.jsonl         the same rows, one file per task (re-weight the mix yourself)
eval_suite_core.jsonl      3,548 rows, Swahili-focused plus 19 regional and major languages for lang_id (fast; run every experiment)
eval_suite_full.jsonl      10,609 rows, every language (run on release candidates)
eval/<task>.jsonl          per-task eval
eval/guard_v0.1_frozen.jsonl  the original 122-row guard eval, unchanged, so you can compare with your first Mwamuzi run
nc/entities_*.jsonl        CC BY-NC data, kept out of train_mix
tasks.json                 registry: questions, source, license, label quality, sizes
stats.json                 label distributions per task
review_sheet_v0.2_authored.csv  367 newly authored rows for native-speaker review

Eval rows carry tags: the task name, src:*, lang:*, core, sw_instructions, plus slices such as domain_boundary, complexity_anchor, jailbreak_neg, harmful_neg, idiom, benign, silver.

Train and evaluate

pip install laya

# 1. baseline (zero-shot)
python tools/run_eval.py --model multilingual --data v0.2/eval_suite_core.jsonl --out preds_base.jsonl

# 2. fine-tune
laya-train --data v0.2/train_mix.jsonl --eval v0.2/eval_suite_core.jsonl --base multilingual \
  --out ./mwamuzi-v0.2 --loss soft-ce --shuffle-options choice --epochs 3

# 3. fine-tuned
python tools/run_eval.py --model ./mwamuzi-v0.2 --data v0.2/eval_suite_core.jsonl --out preds_ft.jsonl
python tools/score.py preds_ft.jsonl --baseline preds_base.jsonl \
  --slices core,sw_instructions,domain_boundary,complexity_anchor,lang:swa,lang:sw --json report.json

score.py reports these metrics:

• choice questions: accuracy and macro-F1.
• yes/no questions: accuracy plus positive-class precision, recall and F1.
• complexity score: exact accuracy, MAE and within-1 accuracy.
• every question: ECE.
• every task: all-questions exact match.

Reading the first Mwamuzi results

• On the 122-row eval, one row is 0.82 points. The +0.82 on PII and harmful is one example, and the jailbreak "drop" is one example. Treat changes of under about 3 points as noise there.
• Safety accuracy hides failures. About 85–94% of guard eval rows are negative, so always answering "no" scores around 90–95%. Judge jailbreak, harmful_request, contains_pii and crisis by F1 and recall on positives; the new scorer reports these.
• Complexity and domain drive the low compound exact match. v0.2 adds explicit tie-break rules to their definitions, 107 new training rows built around domain boundaries and complexity anchors, and eval slices that measure both. Exact match on a 4-level complexity score is harsh, so also track within-1 accuracy.

Design decisions

• No cross-split leakage, including across translations. SIB-200, XCOPA and MultiJail are parallel corpora:
  • SIB-200 uses the same item IDs per split in every language (verified), so its original splits are kept.
  • XCOPA trains only on the val items and evaluates on test.
  • MultiJail is split 80/20 by prompt ID, so no translation of an eval prompt appears in training.
  • Exact duplicates of eval text are also removed from training (15 rows).
• Benign negatives for harm are real Swahili questions from AfriQA, not only statements, so the model can't learn the shortcut "a question means harmful".
• The 8 non-Swahili, non-English MultiJail languages only carry harm_category (every MultiJail prompt is harmful, so they have no benign counterparts). This keeps harmful_request from learning "Bengali or Thai means harmful".
• Swahili instructions on about 35% of public-task rows teach the model to accept Swahili-phrased questions without losing the English path.

Limitations

1. The authored data hasn't been reviewed. This covers guard, scam, toxicity and claims, about 800 unique rows across v0.1 and v0.2. Run review_sheet_v0.2_authored.csv (and the v0.1 sheet) past two native speakers before publishing results on those tasks.
2. The authored eval sets are small (scam 26, toxicity 20, claims 16 unique rows), so treat them as smoke tests until they grow.
3. answer_type labels are rule-based (silver) and heavily skewed toward entity.
4. Licenses:
   • SIB-200 text is CC BY-SA 4.0, so a redistributed derived dataset must keep that license.
   • I took MasakhaNEWS's AFL-3.0 from its dataset card; verify it.
   • MasakhaNER is CC BY-NC, so it stays in nc/.
5. MultiJail contains disturbing prompts (violence, sexual exploitation). That's expected for safety training, but restrict who browses the raw files.
6. SIB-200 repeats the same ~1,000 sentences in 205 languages, which makes it good for cross-lingual transfer but narrow in content.

Rebuild or extend

# clone the source repos into $MWAMUZI_RAW (default /home/claude/raw): dadelani/sib-200, masakhane-io/masakhane-news,
# afrisenti-semeval/afrisent-semeval-2023, cambridgeltl/xcopa, DAMO-NLP-SG/multilingual-safety-for-LLMs,
# masakhane-io/afriqa, masakhane-io/masakhane-ner
python src/build_v02.py && python src/registry.py

To add a task, add its question set to src/task_defs.py (English and Swahili) and a loader block to src/build_v02.py. The writer, leakage filter, tags and stats pick it up automatically.

Next sources to add. These are on Hugging Face, which wasn't reachable from the build environment:

• MASSIVE sw-KE, for intent (60 classes) and assistant domain.
• AfriXNLI and Belebele swh_Latn, for NLI and reading comprehension.
• AfriHate swa, for real hate-speech labels to replace the authored toxicity set.
• KenSwQuAD, for answerability.
• AfriMMLU and Uhura, for knowledge-check tasks.