Does our data improve your model? We measured it.
0.0% baseline · 0.0% scrambled-control · 75.8% fine-tuned — exact-match on unseen tasks.
We took models that score essentially zero on Bantu numeral tasks, fine-tuned them on BantuNomics-generated data (~$5 of compute for the headline run), and graded them on a frozen, fingerprinted held-out exam under all-or-nothing exact-match scoring. A control model trained on the same records with shuffled answers learned the style and still scored 0% — the entire gain is the content being correct.
Per-model results (all runs, nothing omitted)
| Model | Round | Baseline | Control | Fine-tuned | Note |
|---|---|---|---|---|---|
| Qwen3.5-4B | R2 (enriched data) | 0.0% | 0.0% | 75.8% | numerals 95% · arithmetic 87% on unseen items |
| Qwen3.5-4B | R1 | 3.6% | — | 56.8% | arithmetic 24/24; data enrichment → R2 |
| Gemma 4 E4B-it | R1 | 2.9% | — | 18.7% | model choice matters enormously |
| AfriqueQwen-8B | R2 | 0.0% | — | 16.5% | African-language pretraining ≠ Bantu numeracy; best calendar score of any model |
What this teaches about improving your model:
- The data is the dial. Enriching coverage moved 57% → 76% on a harder exam; coverage holes become zeros.
- Content beats fluency. The control proves style-learning scores nothing; only verified forms move the metric.
- Model choice dominates. Same lessons, same budget: 18.7% vs 75.8%.
- Measure before and after. The loop below is free at evaluation tier.
Run the loop on your own model
1. GET /api/tools/benchmark?iso=<your language> # the eval set
2. POST /api/tools/benchmark/score {"iso": "...", "submissions": {"88": "your model's answer", ...}}
3. Train on the generated corpora (Full Subscription: /api/tools/corpus.jsonl|asr|chatml…)
4. POST score again → GET /api/tools/benchmark/runs shows your delta.
MCP: calc_benchmark · calc_score · calc_benchmark_runs (headless).
The interactive dashboard (all 9 runs)
Full reports
Technical report (PDF) Plain-language report (PDF)
Every number, method, command and failure — we publish the incidents too, because for a serious buyer the honesty is the credibility.