Theories and Actions for Reporting and Reducing Achievement Gaps: - PowerPoint PPT Presentation

Theories and Actions for Reporting and Reducing Achievement Gaps: Framework and Examples Brian Gong, Center for Assessment Mary Batiwalla, Tennessee Department of Education Jennifer Stafford, Kentucky Department of Education CCSSO National Conference on Student Assessment June 27, 2018 San Diego

Theories and Actions for Reporting and Reducing Achievement Gaps Brian Gong Drawing on work done with Leslie Keng, Erika Landl, and Juan D’Brot Center for Assessment Presentation in the session on “Theories and Actions for Reporting and Reducing Achievement Gaps: Framework and Examples,” CCSSO National Conference on Student Assessment June 27, 2018 San Diego, CA

Overview Central messages • Reports and reporting will be more effective when the state develops and improves a compelling theory of change that coherently merges theories of action for different agencies/groups. • Reports of achievement gaps /closure must carefully consider the comparison group , the achievement metric , and the gap/closure criteria to be sure they fit the theory of change. • A theory of change should go beyond reporting systems that identify problems to actual provision of effective supports . The state must carefully identify what the state can do to reduce achievement gaps and what others should do. Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 3

Reports and Reporting Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 4

Reporting and a theory of change/use • Reports are a powerful, essential part of a state’s construct definition and theory of change about improving education, e.g., achieving goals related to equity • What is reported (and how and when), to whom, how that information is intended to be interpreted , and especially how that information is intended to be used are all part of the state’s plan for using assessments to improve equitable outcomes • The state should have a detailed, plausible theory of change that specifies actions for results to achieve goals • The state should update its theory of change regularly, based on evidence and sound advice Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 5

Reporting system • Reports – displays of data with intended interpretations and uses • Reporting – actions taken to support intended interpretations and uses (e.g., communication plan to “tell your interpretation,” trainings to help people make appropriate use of data) • Reporting system – like an assessment system, the state should have a reporting system consisting of policies, plans, practices, personnel, and infrastructure to provide capacity and continuity; the effectiveness of the reporting system (e.g., on improving equity) should be evaluated to improve Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 6

Equity and achievement gaps/closure Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 7

Equity and Achievement Gaps • Deciding why attending to equity is important • Defining equity and achievement gaps • Measuring and reporting achievement gaps • Informing reduction of achievement gaps Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 8

Definitions • Excellence – Standards-based excellence ( individual student): A rigorous performance criterion related to comprehensive, relevant content standards – Standards-based excellence ( groups of students): A rigorous performance criterion involving all or virtually all students in an identified student group – Excellence involves both common (all students) and individual aspects – Excellence may also be defined in other ways (e.g., normative) • Equity – Equity as fairness implies that personal or socio-economic circumstances, such as gender, ethnic origin or family background are not obstacles to educational success – Equitable education systems are fair and inclusive and support their students to reach their learning potential without either formally or informally pre-setting barriers or lowering expectations Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 9

Types of Equity and key issues • Equitable to whom – be clear about value choices • Equity in outcomes – Equity in outcomes = converging on Excellence for all? vs. Relative Excellence, growth, etc. • Equity in opportunities – providing access to opportunities designed to lead to more equitable outcomes – Equal vs. equitable • Relationships of equity in opportunities to equity in outcomes – Intermediate outcomes (pre-requisites) – Processes/actions that produce outcomes (how) – Relation with distal outcomes (goals through theory of action) • How time and equity are related – Often concerned about “change over time”; equal time; more time for opportunities Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 10

Equity in outcomes or goals • Comparable in some dimension(s) End point: same End point: different End point: same End time: same End time: same End time: different Slope: different Slope: same, absolute Slope: same, absolute End point: different End time: different Slope: different End point: different End point: % proficient same End point: different End time: same End time: same End time: same Slope: same (same self- Distribution/variance: same, Slope: same (same relative, e.g., “50 th percentile, less, more proportion, e.g., “reduce for own reference group”) non-proficient by 10%”) Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 11

Checking for gap reduction Status/Improvement • Goal-based: comparison of student group performance to goal – Group-based: comparison of one student group to another group – 5 ways a gap may be reduced (status) • Higher, Lower groups both increase; Lower increases more – Higher stays same (ceiling?), Lower increases – Higher declines, Lower stays the same – Higher, Lower both decline; Higher declines more – Any of above, but statistically significant change for time period occurs only for gap reduction – Growth • Cautions when conditional (e.g., value-added) growth – Goal-based – Group-based – How much is “good enough”? How can you tell? • Criteria (e.g., goals) – Statistical tests of differences – Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 12

Equity as factors affecting fairness of opportunity and access • Equity as opportunities and access – Opportunities and access • School offers/student takes • School/district actively supports – Intermediate outcomes (“leading indicators”) • Example: “Pipeline of opportunity” – Schools have 100% of all student groups meet Proficiency goals of scores on state high school math assessments à Schools offer advanced math courses necessary to prepare students for Alg. 2 EOC à Schools have teachers who are effectively teach math à Students enroll in advanced math courses proportional to student group enrollment à Students enroll in preparatory math courses proportional to student group enrollment, by grade 9 à Schools support students to be on-track each grade, etc. • Example: “Early warning and RtI systems” – School monitors student performance and provides effective support well before a student fails the course or drops out (e.g., Attendance/Behavior/Credits) – District provides resources equitably across schools (e.g., effective leadership, effective teachers, coherent curriculum across grades, special services, community engagement) – State helps districts implement district/school/classroom early warning and RtI systems that inform action well before state accountability system’s severe ID & consequences Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 13

Equity as underlying quality of performance • For equity measures to be effective, they must reflect appropriately comparable underlying quality – Be sure users understand any normative/conditional measures – Triangulate multiple measures (especially growth and status; equity process and equity outcomes) to confirm you are getting what you really want • Pay special care to quality of local measures • Pay special care to quality of measures that transform or cross lines from one main user or accountability unit to another (e.g., cross grades; from high to post-high school) (one type of “lagging indicator”) Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 14

Sample report focusing on coherence of quality (e.g., rigor) between local school and external measures Local Measures External Measures Pct High Avg. HS Pct Pct met Pct Avg. Avg. Sch. GPA School enroll in CCR placed in credit credit ID Gradu- college require- remedial hrs hrs (2013-14) course(s) ates ments earned earned (2012-13) (e.g., (met (did not ACT) CCR) meet CCR) 96.3 2.83 70.3 39.0 18.3 10.9 275179 96.1 3.24 63.8 44.9 23.9 3.4 531040 97.1 3.11 61.4 92.2 20.0 10.6 535040 Claim: “If you graduate from our school with good grades, you will be prepared to do well in college.” Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 15