# SYNTHESIS BRIEF: Personalized AI Tutoring at Scale

## Current State Summary

AI tutoring systems have achieved meaningful deployment (Khanmigo reaching ~2M students at $44/student/year; Mindspark operating in India) and show promising but methodologically contested efficacy gains—the oft-cited "14% improvement" from Newark lacks clear operational definitions and baseline context. The field is chasing Bloom's two-sigma benchmark (1-on-1 human tutoring moving average students to the 98th percentile), with a 2024 Stanford meta-analysis suggesting AI systems achieve 0.3-0.5 standard deviations—meaningful but far short of human tutoring. Critical infrastructure constraints (broadband dependency, offline limitations) and unresolved equity questions threaten to replicate rather than close achievement gaps. The evidence base remains weak on long-term retention, transfer effects, and whether gains persist beyond the intervention period.

---

## 5 Most Important Validated Facts

1. **Cost arbitrage is real:** AI tutoring at $44/student/year represents a 95%+ cost reduction versus human tutoring ($40-80/hour), making some form of personalization economically viable at scale for the first time.

2. **Current efficacy falls well short of human tutoring:** AI systems achieve approximately 0.3-0.5 SD gains versus Bloom's 2.0 SD benchmark—roughly 15-25% of the human tutoring effect.

3. **Deployment has reached meaningful scale:** 8,000+ U.S. schools and 2M+ students on Khanmigo alone demonstrates technical feasibility of distribution, though not yet proof of learning outcomes at scale.

4. **Infrastructure dependency creates equity risk:** Systems requiring consistent broadband exclude precisely the populations (rural, low-income) most likely to benefit from tutoring access.

5. **Regulatory frameworks from adjacent domains exist:** FDA's SaMD pathway for digital therapeutics offers a tested model for validating personalized algorithmic interventions across diverse populations.

---

## Top Uncertainties & Resolving Data

| Uncertainty | What Would Resolve It |
|-------------|----------------------|
| **Does the Newark 14% result replicate?** | Independent RCT with state standardized test outcomes, published methodology, and demographic subgroup analysis |
| **Do gains persist after intervention ends?** | 12-24 month longitudinal follow-up studies with control groups |
| **Does AI tutoring close or widen equity gaps?** | Disaggregated efficacy data by income, race, baseline achievement, and infrastructure access |
| **What's the minimum effective "dose"?** | Dose-response studies measuring outcomes against usage intensity and duration |
| **Can offline-capable systems match connected versions?** | Head-to-head trials of Mindspark-style offline models vs. cloud-dependent systems |

**Validate first:** The Newark claim is being cited as foundational evidence. An independent replication with transparent methodology should be the immediate priority before further policy decisions reference it.

---

## Consensus Strategy vs. Competing Strategy

**Consensus Strategy:** Hybrid deployment—AI tutoring as supplement to (not replacement for) classroom instruction, targeting high-frequency practice domains (math facts, reading fluency) where immediate feedback loops show strongest effects. Scale through district partnerships with subsidized pricing; invest in teacher training for integration.

**Competing Strategy:** Leapfrog model—deploy directly to underserved populations via mobile-first, offline-capable platforms (Mindspark approach), bypassing institutional adoption bottlenecks. Accepts lower per-session efficacy in exchange for dramatically higher reach and usage frequency. Prioritizes access over optimization.

**The tension:** Consensus strategy optimizes for measurable outcomes in existing systems; competing strategy optimizes for reaching students currently outside any system. Evidence is insufficient to declare a winner—both need parallel investment.

---

## Key Milestones

### 6 Months
- Independent replication study of Newark/Khanmigo results initiated with pre-registered methodology
- At least one major platform releases disaggregated efficacy data by demographic subgroups
- Offline-capable feature parity achieved by one major U.S. platform

### 12 Months
- First longitudinal data (12+ months post-intervention) published on retention of learning gains
- Regulatory clarity: either voluntary efficacy standards adopted by major platforms or state-level requirements proposed
- Cost per student drops below $30/year for at least one validated system

### 24 Months
- Meta-analytic evidence base includes 10+ RCTs with standardized outcome measures
- Clear dose-response relationship established (minimum usage for meaningful effect)
- At least one system demonstrates efficacy gains >0.7 SD in controlled conditions, narrowing gap to human tutoring

---

**Evidence Quality Assessment:** Current evidence is **weak-to-moderate**. Headline claims (14% improvement) lack methodological transparency. The 0.3-0.5 SD meta-analytic finding is more credible but aggregates heterogeneous interventions. No long-term retention data exists. Funders and policymakers should treat current results as promising signals requiring validation, not proven interventions ready for universal deployment.