Phone-based Learning Assessments: Lessons from Botswana Noam - PowerPoint PPT Presentation

Phone-based Learning Assessments: Lessons from Botswana Noam Angrist

Prior to Covid-19: Scaling Up Teaching at the Right Level in Botswana Reached 15% of schools cumulatively

COVID-19 Response: Low-Tech Education Low-tech: calls, Parents as teachers at the Rapid trial, data in Immediate policy implications for texts & radio household monthly cycles government & teachers

Rapid Trial ~13% initial dropout ~71% consent & interest Phone assessment

Distribution of Respondent Schools

Sample of ASER test used in Botswana

Sample Phone Call Script

Training

Example Strategies Lessons Specific Strategies • 1. Protect Children E.g. low-stakes, ask parent for privacy • 2. Test reliability of measures Build on prior assessments (e.g. ASER), include randomized sets of problems 3. Keep instructions simple & practice first • 4. Some assessments more conducive than others e.g. reading need to combine with text • 5. Keep it short 20 minutes of call-time maximum 6. Experiment to get phone response take-up 7. Establish rapport • 8. Choose cost-effective approaches Two-way texting logistically harder and lower take-up so did phone call • 9. Account for sample bias Demographic info and sample comparison

Learning Level Comparison

Table 1: Sample Characteristics Comparison

Cost-effectiveness Test Cost per Child PIRLS 2011 Botswana $62.5 Phone Assessment $4.40

Future • More validation: o Random set of problems for internal validity checks o Non-cognitive question to tease out effort from cognitive skills o Follow-on waves of data collection and multi-context assessment o Potential to complement existing testing infrastructure (PISA,TIMSS, SACMEQ, PASEC, ASER, Uwezo, EGRA etc) – potentially for higher frequency, low-stakes diagnostics o BMJ Global Health paper on measuring learning: o Angrist, Bergman, Evans, Hares, Jukes, Letsomo . 2020. “Practical lessons for phone -based assessments.” BMJ Global Health , forthcoming. o Forthcoming RCT results: o Angrist, Bergman, Brewster, Matsheng . 2020. “ Stemming Learning Loss During the Pandemic: A Rapid Randomized Trial of a Low- Tech Intervention in Botswana.”