Final Evaluation Results of The Milwaukee Community Literacy Project/ SPARK Early Literacy Curtis Jones, SREed, University of Wisconsin, Milwaukee Michael Christian, Education Analytics Andrew Rice, Education Analytics March, 3 rd , 2016 Presented to the Annual Meeting of the Society for Research on Educational Effectiveness, Washington DC Project funded by a Department of Education Investing in Innovations (i3) grant. Contract number U396C100694. 1 SPARK/MCLP Evaluation
Literacy Context in Milwaukee The need for increased literacy development opportunities for Milwaukee students is urgent. According to the 2013 National Assessment of Educational Progress (NAEP) 15% of 4th grade MPS students were At or Above Proficient in reading. This need is even more pronounced for low-income and minority students. • 38% of 4th grade, White MPS students were proficient in reading, compared to 9% of Black and 14% of Hispanic students. • 11% of 4th grade low-income (free/reduced lunch participants) MPS students were proficient in reading, compared to 39% of non-low-income students. 2 SPARK/MCLP Evaluation
History of SPARK Early Literacy 2005 - SPARK was created to address this need by Boys & Girls Clubs of Greater Milwaukee and piloted at one site in Milwaukee Public Schools (MPS). 2006 - SPARK was expanded to three MPS schools with funding from the United Way and AmeriCorps. 2010 - SPARK received a Department of Education investing in innovation (i3) grant award to expand to seven more schools (10 total). 3 SPARK/MCLP Evaluation
Program: SPARK Early Literacy Logic Model The need for increased literacy opportunities in the Milwaukee area is urgent. According to the 2013 National Assessment of Educational Situation: Progress (NAEP) 15% of 4th grade MPS students were At or Above Proficient in reading. This need is even more pronounced for low-income and minority students. 38% of 4th grade, White MPS students were proficient in reading, compared to 9% of Black and 14% of Hispanic students. 11% of 4th grade low-income (free/reduced lunch participants) MPS students were proficient in reading, compared to 39% of non-low-income students. Inputs Outputs Outcomes Activities Participation Short Medium Long After-School Boys & Girls Clubs Club after-school Students participate in staff and programming academic and Club programming (e.g. KidzLit) enrichment throughout elementary programming school. Improved reading (Running Records) Tutoring SPARK site manager One-on-one tutoring is Students receive 30 Sustained improvement (certified teacher), provided during the minutes of tutoring, in reading achievement; Improved literacy Improved reading tutors, SPARK room, school day in a print- three times per week, School success (PALS Assessment) achievement materials (bins) rich environment. for up to two years. Improved school Family performance Family engagement Family events; parents Monthly family events; manager, parent contacts; newsletters; 2 home visits; 2 liaisons, parents home visits contacts per month; Families more engaged monthly newsletters in and knowledgeable about student's Support/Collaboration education. Project director, Ongoing SPARK PD & Tutors observed 1x/ district liaison, support month; Staff receive teachers, evaluation, PD. school and district Rev: 12 16 15 leadership Ongoing mtgs with Collaboration teachers, district and school leadership Assumptions Context SPARK adds capacity to a school by supplying tutors and a parents liaison to work Many districts face staffing and budget shortages that prevent them from with students. using other reading interventions. Typical literacy interventions only focus on skill acquisition and ignore the reasons The RTI model does not work when 90% of students are below proficiency in reading. why students fall behind. The benefits observed in other literacy interventions have typically faded over time. Chronic absenteeism (missing more than 10% of school days) is a serious Boys & Girls Clubs is in a unique position, however, to continue to work with program in MPS and other urban districts. Many parents do not understand students and families beyond SPARK. the importance of early grade participation. 4 SPARK/MCLP Evaluation
Collaboration and Support Support/Collaboration Project director, Ongoing SPARK PD Tutors observed 1x/ district liaison, & support month; Staff receive teachers, PD. evaluation, school and district Ongoing mtgs with Collaboration leadership teachers, district and school leadership 5 SPARK/MCLP Evaluation
Inputs After-School Certified teacher in each site supports and supervises a cadre of Boys & Girls Clubs about six tutors. staff and programming (e.g. Tutors include AmeriCorps members, UWM preservice KidzLit) teachers, and a small number of community members. Tutoring Tutoring is done in a print- rich “SPARK room” SPARK site Each site also has a parent liaison who would work with manager (certified teacher), tutors, SPARK families. SPARK room, materials (bins) Family Family engagement manager, parent liaisons, parents 6 SPARK/MCLP Evaluation
Activities - Tutoring Students are pulled out of non-core classes and taken to Outputs SPARK room three times per week for up to two years. Activities Participation Starts with familiar activity. Club after-school Students participate in academic and Club programming Running Records assessment every third session. enrichment throughout elementary programming school. Word play - Students receive instruction using Word Sorts and Making Words. Word Sorts involve students sorting words into One-on-one tutoring is Students receive 30 categories to increase their understanding of sounds and letters. provided during the minutes of tutoring, Making Words involves students using letters to make words so school day in a print- three times per week, students learn how the sounds of language are put together. rich environment. for up to two years. Reading a book at instructional level. Family events; parents Monthly family events; Writing sentences - Elkonin boxes are a central piece of contacts; newsletters; 2 home visits; 2 home visits contacts per month; SPARK writing and used to help students encode words. monthly newsletters End with tutor read aloud. 7 SPARK/MCLP Evaluation
Activities – Family Engagement The goals of family engagement is to keep families aware of Outputs student progress in SPARK, help families promote literacy at Activities Participation home, and address any school attendance issues that arise during the program. Club after-school Students participate in academic and Club programming Monthly family events, enrichment throughout elementary programming school. 2 home contacts per month (email, in person, or by phone), 1 home visit per year, usually over the summer, One-on-one tutoring is Students receive 30 provided during the minutes of tutoring, school day in a print- three times per week, Monthly newsletters sent home. rich environment. for up to two years. Family events; parents Monthly family events; contacts; newsletters; 2 home visits; 2 home visits contacts per month; monthly newsletters 8 SPARK/MCLP Evaluation
Evaluation Design The primary goal of the evaluation was to design a study that would meet What Works Clearinghouse (WWC) design standards without reservations. • Studies have to use a randomized-control framework to qualify. 576 students were randomly assigned to either the control (290) or SPARK (286) group by the evaluation team. Assignments were stratified by school and grade level within schools. ELL and IEP students were not eligible for study. • Studies have to use valid and reliable outcome measures. NWEA Measures of Academic Progress (MAP) Reading Benchmark Assessment, Phonological Literacy Screening (PALS), Regular-school-day attendance. • Studies have to meet WWC attrition and differential attrition standards. 9 SPARK/MCLP Evaluation
Overall Selection & Attrition Results 33.9% - 36.3% Overall Attrition Rate Missing Data Final sample 576 Students 389 (187 of b/w 367 selected Moved) and 381 10 SPARK/MCLP Evaluation
Differential Attrition Results 1.9% to 2.3% Differential Attrition Rate SPARK Participants 32.9% - 35.3% attrition b/w 185-192 (3- 286 Students 195 (91Moved) 10 missing data) Control Students 34.8% - 37.2% attrition b/w 182-189 (5- 290 Students 194 (96 Moved) 12 missing data) 11 SPARK/MCLP Evaluation
What Works Clearinghouse (WWC) Attrition Standards 12 SPARK/MCLP Evaluation
Final Sample Control SPARK Total K-1 st Grade Level 72 66 138 1 st -2 nd 63 74 137 2 nd -3 rd 59 55 114 Race/Ethnicity Black 149 146 295 Hispanic 29 34 63 Other 16 15 31 Gender Female 96 100 196 Male 98 95 193 F/R Lunch Eligible No 9 8 17 Yes 185 187 372 IEP No 182 184 366 Yes 12 11 23 Total 194 195 389 13 SPARK/MCLP Evaluation
Recommend
More recommend
