USAID’s Early Grade Reading and Early Grade Math Impact Evaluations: Lessons Learned National Education Week Subtitle placeholder Reforming the Education Sector for Effective Service Delivery: Embracing Innovations Monica Gadkari Chief of Party, USAID Partnership for Education, Evaluating Systems mgadkari@socialimpact..com
Presentation Outline This presentation looks at lessons learned with regards to teacher performance from USAID’s Early Grade Reading and Math Programs : Program Designs Understanding the Impact Evaluation Impact and outcome results Fidelity of implementation Recommendations FOOTER GOES HERE 9/5/19 2
PROGRAM DESIGNS Numeracy Pilot Early Grade Reading Program Revised syllabus with greater emphasis on Use of Ghanaian language and focus on conceptual understanding and mathematical phonics versus “whole language” IMPROVED reasoning approach to literacy acquisition PEDAGOGY Ghanaian language materials including Teacher resource guide with weekly work scripted lesson plans, pupil workbooks, schemes, games and activities, performance supplementary readers, and assessment standards, and assessment tasks MATERIALS tools Training of Math Coaches who facilitate Teachers trained on use of new materials INSET and weekly learning circles to help through training events and weekly teachers put the resources into practice school-based INSET meetings TRAINING Math Coaches and Head Teachers to provide Head Teachers, Curriculum Leads, and regular classroom observation and Circuit Supervisors regularly observe and COACHING & coaching/feedback coach teachers on instructional practices MONITORING 9/5/19 3
EVALUATION QUESTIONS Impact / Goal Questions After 1 year, to what extent does the Numeracy Pilot improve early grade mathematics performance for P1 and P2 learners, as Math measured by the Early Grade Mathematics Assessment (EGMA) and select subtests of the Ghana Early Numeracy Assessment (GENA)? After 2 years, to what extent does the Early Grade Reading Reading Program improve P1 and P2 reading skills, as measured by the Early Grade Reading Assessment (EGRA)? 9/5/19 4
EVALUATION RESEARCH DESIGN Math Randomized Controlled Trial (RCT) Sampling frame: all schools in Shai Osudoku and New 60 schools Juanbeng (121 total) randomly assigned to numeracy pilot 61 schools randomly assigned to control 9/5/19 5
EVALUATION RESEARCH DESIGN Reading Quasi-Experiment Using Statistical Matching Intervention Comparison FOOTER GOES HERE 9/5/19 6
PROGRAM IMPACT MATH Math Pupil Mathematics Performance EGRA: Procedural understanding of GENA: Conceptual Understanding of math math 3.8% increase in P1 5.3% increase in 17.3% increase in P1 18.9% increase P2 in P2 Small reduction in zero scores Small reduction in zero scores FOOTER GOES HERE 9/5/19 7
PROGRAM IMPACT READING 18.54 4.72 4.04 5.17 6.17 letter Percent words per words per Percent sounds per minute minute minute Pre-Reading Initial Reading Fluency/Comprehension kɛw rɔm mɛn nuw lim lɔm gɔm lam mun Reading Comprehension Non-Word Reading Listening Oral Reading Fluency s y m o s y m o Comprehension Abasɛm tiawa bi ni. Mepɛ sɛ wokenkan no dennennen, ne ntɛmntɛm ma me. Ɛ y m N Ɛ y m N Wokenkan wie a, mebisabisa nsɛm bi afa nea woakenkan no ho. Meka sɛ “Fi Ase” a u k Ɔ a u k Ɔ a kenkan abasɛm no sɛnea wubetumi biara. Wudu asɛmfua bi so na wunnim a, kɔ asɛmfua foforo so. Letter Sounds FOOTER GOES HERE 9/5/19
PROGRAM IMPACT READING 40 The average gains are not 33.76 35 representative of all 30 students. 25 22.72 The most significant gains 20 exist amongst the top 10% 15 and top 25% of students. 8.64 10 6.9 5 The driver of the low 1.54 0.9 0 averages are zero scorers Average Top 25% Top 10% Control Treatment
TEACHER PERFORMANCE MATH Math 29% more likely to ask students to find and share problem solving strategies The following observed changes to teacher performance show 15% more likely to engage in mathematical greater use of teaching communication approaches that promote a conceptual understanding of 14% more likely to provide students opportunities to use mathematical reasoning math 10-15% more likely to use active learning techniques in classroom Reduced frequency of assessment but greater response to assessment data 9/5/19 10
TEACHER PERFORMANCE READING Reading 7% increase in time spent on learning activities Most outcomes surrounding changes in 28% increase in coaching and mentoring pupil and teacher behaviour were 20% increase in coaching based on positive observation Reduced frequency of assessment Improvement in implementation of 4 out of 5 teaching best practices 9/5/19 11
FIDELITY OF IMPLEMENTATION MATH In terms of use, lesson plans and Fully Implemented Partially teacher guides implemented • Training of head teachers were widely • Training of math coaches • Math coaches provided only used, while • Provision of Materials 20% of planned INSET training classroom • Math coaches, head teachers, and circuit supervisors provided materials, even support at 30% dosage when present, • Supplementary materials not were used developed consistently inconsistently 9/5/19 12
FIDELITY OF IMPLEMENTATION READING About 80% of On-Track Component treatment Lesson CS Monitoring schools in a Performance Component Component state of “full” or “near full” SBI Coaching Pupil Materials Component Component implementation fidelity on HT/CL Coaching Teacher Materials multiple Component Component components. Teacher In- Supplemental Training Materials Component Component
PREDICTORS OF PERFORMANCE MATH AND READING Predictor Outcome Predictor Outcome Pupil always hungry High fidelity to ▲ at the start of program ▼ school Enthusiasm for Teacher punishes ▲ reading and math pupil for poor ▼ performance ▲ Female student School is in rural ▼ locality Higher percent of reading teachers ▲ Low language ▼ who are female match FOOTER GOES HERE 9/5/19 14
PROGRAM PREDICTORS OF SUCCESS CS Monitoring The individual SBI program component most significantly Obs. & Coaching associated with Reading Instr. Training program impact is lesson performance Sup. Materials the extent to which Pupil Workbook teachers follow the Lesson Performance scripted lesson plans and engage their Pace students in doing so -0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Letter Sounds ORF
PROGRAM PREDICTORS OF SUCCESS Effect Size ORF 0.9 Overall fidelity to 0.8 the program was 0.7 essential. The effect 0.6 size of the program 0.5 schools with low or partial fidelity were 0.4 significantly lower 0.3 than those with near 0.2 full or full FOI 0.1 0 Low FOI Partial FOI Near-Full Full FOI FOI
Outcomes RECOMMENDATIONS Ø Program lesson plans and teacher guides are helpful to teachers and are associated with success Ø For full programmatic scale up, MOE and GES management should integrate more accountability mechanisms that ensure greater fidelity of implementation Ø Teacher training curriculum should encourage teachers to give positive rather than negative reinforcement to students. Ø Research how to overcome the gap between the large conceptual gains and the smaller procedural gains we see on math performance 9/5/19 17
Thank you
Recommend
More recommend
