NUMBERS fine-tuning — model comparison

Does BantuNomics data make models better at Bantu numeracy — and how much? Deterministic accepted-form scoring on held-out cases. Regenerate with python scripts/build_dashboard.py after each eval.

Best fine-tuned accuracy
75.8%
Qwen 3.5-4B-Base · R2
Round-1 lift (Qwen)
16×
3.6% → 56.8%
Arithmetic (held-out, R1)
24/24
compositional numerals — not memorizable
Modal spend
≈ $41 total
acct 1: $30 (incl. ~$20 timeout lesson) · acct 2: ≈ $11
Baseline (stock model)Fine-tuned (BantuNomics data)Control (shuffled targets)

Round 1 — 2026-08-05

139 held-out cases (round-1 split) · 3 epochs on 1,027 records · ~$1–2/run

0%25%50%75%100%Gemma 4 E4B-it — Baseline (stock model)2.9%Gemma 4 E4B-it — Fine-tuned (BantuNomics data)18.7%Qwen 3.5-4B-Base — Baseline (stock model)3.6%Qwen 3.5-4B-Base — Fine-tuned (BantuNomics data)56.8%

numerals

0%25%50%75%100%Gemma · Baseline6.2%Gemma · Fine-tuned35.9%Qwen · Baseline7.8%Qwen · Fine-tuned60.9%

arithmetic

0%25%50%75%100%Gemma · Baseline0.0%Gemma · Fine-tuned0.0%Qwen · Baseline0.0%Qwen · Fine-tuned100.0%

noun class

0%25%50%75%100%Gemma · Baseline0.0%Gemma · Fine-tuned4.5%Qwen · Baseline0.0%Qwen · Fine-tuned54.5%

calendar

0%25%50%75%100%Gemma · Baseline0.0%Gemma · Fine-tuned12.5%Qwen · Baseline0.0%Qwen · Fine-tuned0.0%

tutor

0%25%50%75%100%Gemma · Baseline0.0%Gemma · Fine-tuned0.0%Qwen · Baseline0.0%Qwen · Fine-tuned30.8%
Table view — all numbers
ModelConditionOverallnumeralsarithmeticnoun_classcalendartutor
Gemma 4 E4B-itBaseline (stock model)2.9%4/640/240/220/160/13
Gemma 4 E4B-itFine-tuned (BantuNomics data)18.7%23/640/241/222/160/13
Qwen 3.5-4B-BaseBaseline (stock model)3.6%5/640/240/220/160/13
Qwen 3.5-4B-BaseFine-tuned (BantuNomics data)56.8%39/6424/2412/220/164/13
Sample predictions — Qwen 3.5-4B-Base Baseline (stock model)
PromptPredictionAccepted
What number is 'tharo' in Sesotho? Answer with digits only.33
Write the number 3 in Silozi.In Silozi, the number 3 is written as **"tatu"**.ze taalu
What number is 'kunye' in siSwati? Answer with digits only.The number "kunye" in SiSwati means "one". Therefore, the an1
Compute and express in iciBemba: 3 + 4 = 7In the **IciBemba** language, the sentence "3 + 4 = 7" wouldfitatu sanshapo fine ni cine lubali
Write the number 13 in iciBemba.In the Bemba language, the number 13 is expressed as **"mwanikumi limo na fitatu, ikumi na fitatu
Sample predictions — Qwen 3.5-4B-Base Fine-tuned (BantuNomics data)
PromptPredictionAccepted
What number is 'tharo' in Sesotho? Answer with digits only.33
Write the number 3 in Silozi.ze taaluze taalu
What number is 'kunye' in siSwati? Answer with digits only.11
Compute and express in iciBemba: 3 + 4 = 7fitatu sanshapo fine ni cine lubalifitatu sanshapo fine ni cine lubali
Write the number 13 in iciBemba.ikumi na mutandaikumi limo na fitatu, ikumi na fitatu

Round 2 — 2026-08-07

400-case stratified sample of the 2,773-case frozen test split · lean pair: 2 epochs on 5,840 stratified records (ALL calendar+tutor kept; engine = live 10-language repo) vs its shuffled-target CONTROL twin — same tokens, destroyed mapping

0%25%50%75%100%Qwen 3.5-4B-Base — Baseline (stock model)0.0%Qwen 3.5-4B-Base — Fine-tuned (BantuNomics data)75.8%Qwen 3.5-4B-Base — Control (shuffled targets)0.0%afriqueqwen_8b — Baseline (stock model)0.0%afriqueqwen_8b — Fine-tuned (BantuNomics data)16.5%

numerals

0%25%50%75%100%Qwen · Baseline0.0%Qwen · Fine-tuned94.6%Qwen · Control0.0%afriqueqwen_8b · Baseline0.0%afriqueqwen_8b · Fine-tuned35.1%

arithmetic

0%25%50%75%100%Qwen · Baseline0.0%Qwen · Fine-tuned87.1%Qwen · Control0.0%afriqueqwen_8b · Baseline0.0%afriqueqwen_8b · Fine-tuned9.5%

noun class

0%25%50%75%100%Qwen · Baseline0.0%Qwen · Fine-tuned52.2%Qwen · Control0.0%afriqueqwen_8b · Baseline0.0%afriqueqwen_8b · Fine-tuned23.9%

calendar

0%25%50%75%100%Qwen · Baseline0.0%Qwen · Fine-tuned13.3%Qwen · Control0.0%afriqueqwen_8b · Baseline0.0%afriqueqwen_8b · Fine-tuned30.0%

tutor

0%25%50%75%100%Qwen · Baseline0.0%Qwen · Fine-tuned55.0%Qwen · Control0.0%afriqueqwen_8b · Baseline0.0%afriqueqwen_8b · Fine-tuned0.0%
Table view — all numbers
ModelConditionOverallnumeralsarithmeticnoun_classcalendartutor
Qwen 3.5-4B-BaseBaseline (stock model)0.0%0/740/2100/460/300/40
Qwen 3.5-4B-BaseFine-tuned (BantuNomics data)75.8%70/74183/21024/464/3022/40
Qwen 3.5-4B-BaseControl (shuffled targets)0.0%0/740/2100/460/300/40
afriqueqwen_8bBaseline (stock model)0.0%0/740/2100/460/300/40
afriqueqwen_8bFine-tuned (BantuNomics data)16.5%26/7420/21011/469/300/40
Sample predictions — Qwen 3.5-4B-Base Baseline (stock model)
PromptPredictionAccepted
Compute and express in isiZulu: 2 × 8 = 162 × 8 = 16 In isiZulu: **2 × 8 = 16** *(The mathematicakubili uphindaphinda isishiyagalombili kulingana neshumi nes
Compute and express in Kinyarwanda: 9 × 17 = 153**Kinyarwanda:** **9 × 17 = 153** **Ibisobanuro:** - **9icyenda kuba cumi na karindwi bingana na ijana mirongo itanu
Compute and express in Silozi: 19 + 18 = 37In Silozi, the numbers are expressed as follows: - 19 = **1ze lishumi ni ze ketalizoho ka zene kukopanya ze lishumi ni
Compute and express in isiXhosa: 8 × 14 = 112**Isixhobo:** 1. **8 × 14 = 112** 2. **Isixhobo:** 8sibhozo phindaphinda neshumi elinane kulingana nekhulu eline
Compute and express in Chitonga: 15 ÷ 12 = 1In Chitonga, the expression "15 ÷ 12 = 1" can be translated kumi atosanwe njizya kumi atobile cikozyene komwe, kusyala t
Sample predictions — Qwen 3.5-4B-Base Fine-tuned (BantuNomics data)
PromptPredictionAccepted
Compute and express in isiZulu: 2 × 8 = 16kubili uphindaphinda isishiyagalombili kulingana neshumi neskubili uphindaphinda isishiyagalombili kulingana neshumi nes
Compute and express in Kinyarwanda: 9 × 17 = 153icyenda kuba cumi na karindwi bingana na ijana mirongo itanuicyenda kuba cumi na karindwi bingana na ijana mirongo itanu
Compute and express in Silozi: 19 + 18 = 37ze lishumi ni ze ketalizoho ka zene kukopanya ze lishumi ni ze lishumi ni ze ketalizoho ka zene kukopanya ze lishumi ni
Compute and express in isiXhosa: 8 × 14 = 112sibhozo phindaphinda neshumi elinane kulingana nekhulu elinasibhozo phindaphinda neshumi elinane kulingana nekhulu eline
Compute and express in Chitonga: 15 ÷ 12 = 1kumi atosanwe njizya kumi atobile cikozyene komwe, kusyala tkumi atosanwe njizya kumi atobile cikozyene komwe, kusyala t
Sample predictions — Qwen 3.5-4B-Base Control (shuffled targets)
PromptPredictionAccepted
Compute and express in isiZulu: 2 × 8 = 16kumi na tisa gawanya kumi na saba ni moja, salio mbili user kubili uphindaphinda isishiyagalombili kulingana neshumi nes
Compute and express in Kinyarwanda: 9 × 17 = 153kumi na tisa gawanya kumi na saba ni moja, salio mbili user icyenda kuba cumi na karindwi bingana na ijana mirongo itanu
Compute and express in Silozi: 19 + 18 = 37kumi na tisa gawanya kumi na saba ni moja, salio mbili user ze lishumi ni ze ketalizoho ka zene kukopanya ze lishumi ni
Compute and express in isiXhosa: 8 × 14 = 112kumi na tisa gawanya kumi na saba ni moja, salio mbili user sibhozo phindaphinda neshumi elinane kulingana nekhulu eline
Compute and express in Chitonga: 15 ÷ 12 = 1kumi na tisa gawanya kumi na saba ni moja, salio mbili user kumi atosanwe njizya kumi atobile cikozyene komwe, kusyala t
Sample predictions — afriqueqwen_8b Baseline (stock model)
PromptPredictionAccepted
Compute and express in isiZulu: 2 × 8 = 16## Step-by-step Solution Problem to solve: $2\cdot 8$ 1 Mkubili uphindaphinda isishiyagalombili kulingana neshumi nes
Compute and express in Kinyarwanda: 9 × 17 = 153### 17.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1icyenda kuba cumi na karindwi bingana na ijana mirongo itanu
Compute and express in Silozi: 19 + 18 = 37### 19 + 18 = 37 19 + 18 = 37 19 + 18 = 37 19 + 18 = 37 19 ze lishumi ni ze ketalizoho ka zene kukopanya ze lishumi ni
Compute and express in isiXhosa: 8 × 14 = 112## Step-by-step Solution Problem to solve: $8\cdot 14$ 1 sibhozo phindaphinda neshumi elinane kulingana nekhulu eline
Compute and express in Chitonga: 15 ÷ 12 = 1### 15 ÷ 12 = 1.25 #### Division of 15 by 12 (15 ÷ 12) 15 kumi atosanwe njizya kumi atobile cikozyene komwe, kusyala t
Sample predictions — afriqueqwen_8b Fine-tuned (BantuNomics data)
PromptPredictionAccepted
Compute and express in isiZulu: 2 × 8 = 16kubili uphindaphinda isishiyagalombili kulingana neshumi neskubili uphindaphinda isishiyagalombili kulingana neshumi nes
Compute and express in Kinyarwanda: 9 × 17 = 153icyenda kuba cumi na karindwi bingana na ijana mirongo itanuicyenda kuba cumi na karindwi bingana na ijana mirongo itanu
Compute and express in Silozi: 19 + 18 = 37ze lishumi ni ze ketalizoho ka zene kukopanya ze lishumi ni ze lishumi ni ze ketalizoho ka zene kukopanya ze lishumi ni
Compute and express in isiXhosa: 8 × 14 = 112isibhozo phindaphinda neshumi elinane kulingana nekhulu elinsibhozo phindaphinda neshumi elinane kulingana nekhulu eline
Compute and express in Chitonga: 15 ÷ 12 = 1kumi atosanwe njizya kumi atobile cikozyene komwe, kulasambwkumi atosanwe njizya kumi atobile cikozyene komwe, kusyala t

Rounds use different frozen test sets, so bars are comparable WITHIN a round, never across rounds (separate axes by design). Control = same inputs & token budget, target mapping destroyed — if it matches Baseline while Fine-tuned clears both, the lift is the data's content. Source: results/evals/*.json · plan: ENHANCED_FINETUNE_PLAN_v2.md