What is School Quality? The Potential of EWS to Inform School - PowerPoint PPT Presentation

What is School Quality? The Potential of EWS to Inform School Accountability Under ESSA Presented at NCSA June 28, 2018 by Jurupa Unified School District Presentation Team Jurupa Valley, California

Jurupa Unified School District K-12 Enrollment = 19,112 SED = 82% English Learners = 31% Hispanic Latino = 86% White = 9% African American = 2% Asian = 1%

Presentation Team Elliott Duchon, Superintendent JUSD, educhon@jusd.k12.ca.us Jay Trujillo, Director, Secondary Education JUSD, jtrujillo@jusd.k12.ca.us Pete Goldschmidt, PhD, Research Consultant, pete.goldschmidt@csun.edu Sandy Sanford, EdD, Assessment Consultant, sandy@youasksandy.com Heather Goodwin, Assessment Consultant, good1hlg@gmail.com

Presentation Plan • Slides (PPT posted on NCSA 2018 web site) • Pete Videos available by requesting from Sandy’s Email sandy@youasksandy.com • EWS vs. EEWS

What Is School Quality: The Potential of EEWS to Inform School Accountability Under ESSA Pete Goldschmidt, PhD. National Conference on Student Assessment June 28, 2018

Pete Video Here

Definition Extended Early Warning System • Most Early Warning Systems predominantly use data from middle and or high school. • Extended refers to both greater depth and breadth in data utilized. • Using data from elementary school • Using student responses to a school bonding survey.

EEWS—Behind the Scenes User Interface Information Processing Flags Modeling Modeling Modeling Module Modeling Module Modeling Module EEWS Modeling Module Modeling Module Modeling Module Data Base Module Module VAM

What does EEWS User Interface Look Like?

Four Student Factors • Student Background ü Student Information System • Academics ü Assessment Data • Behavior ü Annual Student Survey • Social Emotional

Four Student Factors—Student Background • Characteristics such as SES, SWD, Language, etc. • Family Support • My family expects me to attend college • An adult at home asks about my homework and classwork.

Four Student Factors—Academics • Assessment Results • English Language Development • Course Credits • F Grades • On Track

Four Student Factors—Behavior • Absences • Absences first month of school • Discipline* • Suspension

Four Student Factors—Social Emotional • Locus of Control • School Bonding • School Culture

Four Student Factors—Social Emotional • Locus of Control • “ Locus of control (LoC) orientation is a belief about whether the outcomes of actions are contingent on what we do (internal control orientation) or on events outside our personal control (external control orientation)." (Zimbardo, 1985). Some evidence suggests that context influences LoC and that LoC is malleable. • When I study hard, I do well. • I am good at math • My success in math is completely up to me. • I can be myself at this school.

Four Student Factors—Social Emotional • School Bonding • My teacher cares about me • I have friends who help me at school • I feel respected by other students

Four Student Factors—Social Emotional • School Culture • I see my principal in my classroom • I feel safe at this school • I have been called mean names, made fun of, hit, or pushed at school

Flags • Markers pre-programmed on the EEWS database to compile what is determined to be meaningful information—30 so far • For example, ü 5 or more absences during first 30 school days ü Locus of Control ü School Year Off Track ü Prior Year Fs ü Middle to High School Absence Change ü Teacher Cares

How can we check to see whether or not any particular Flag is ACCURATE or MEANINGFUL?

Pete Video Here

Verifying Flags • Literature generally applies a 75/10 “rule”. • 75% students with the outcome have the flag. • 10% of students identified by flag, have the outcome. • Relative Risk. • Conditional Effect. • Health science literature slightly different, yet similar, approach. • Sensitivity – Ability to test positive if have outcome (10) • Specificity- Ability to test as outcome “free” • PPV – Probability of outcome when test positive (75) • If around 90% very strong Flag. • NPV – Probability do not have outcome with no Flag

Verifying Flags • Early Warning Flags generally do not perform as well as medical tests (although many medical tests do not perform nearly as well as most people think). • The relative importance of misclassifications depends on existing/intended/(to be) developed strategies. • e.g., School-wide strategies impact all students – both those for whom it is intended and others Vs. Specific student centered strategies .

Classroom Impact on Math Scores

School Bonding and School Performance – Grade 6

Percent of Students at each School with Flag 1 0 0 % 9 0 % 8 0 % 7 0 % 6 0 % 5 0 % 4 0 % 3 0 % 2 0 % 1 0 % 0 % 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 0 5 0 0 5 0 0 0 9 0 l 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 0 0 1 0 0 1 0 0 1 3 0 2 0 a t 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 3 3 3 4 4 4 4 5 5 6 o 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 T 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 First3 5 Flag 5 T otAb sF l g An y_ Di sc Flag Susp e n _ FLG FsOff_ T rac k T y Of f_ T ra ck iStu dy_ Flag iR e ad_ Flag iM a t h_ F l a g Sc h oo l_ F l a g LoC _ Flag An y_ Flag

EEWS Supporting Jurupa Programs

Di District Crisis The # of Students Failing and NOT Graduating was Growing

Accelerating Academic Achievement th Gr 9 th Grade AAA AAA Ac Academy

Big Deal • Unprecedented • Unpopular • Costly

Evaluation of the AAA Program In real-time & On-going

EEWS Support of Evaluation • Created Meaningful Comparison Group • Determined most effective entrance criteria • Determine early grades predictors of AAA candidates • Measured long-term effects (does the program have legs?)

AAA Evaluation Results • In all measured area AAA students outdid control group • Attendance • Discipline • Academics • BUT, the improvement gap narrowed after the treatment year. This led to improved post treatment support .

N O M OR E D .R .I .P .

Pete Video Here

Summary • Identify relevant outcomes • Identify potential malleable indicators that may be related to outcomes. • Examine extent to which indicators differentiate between students with and without the flag. • Determine whether and how to combine various indicators into a composite. • Examine extent to which school level composite works within accountability system (see Handbook).

Questions?