Comparability Evaluation Options for the Innovative Assessment and Accountability Demonstration Authority Susan Lyons & Scott Marion Center for Assessment CCSSO’s NCSA 2017 June 28, 2017
Project Goals 1.Articulate a framework for comparability for the Demonstration Authority under ESSA 2.Expand the comparability options in draft regulations 3.Support states in planning innovative assessment pilots Thank you to the William and Flora Hewlett Foundation funding of this work. Lyons & Marion_Comparability Options for the Innovative Pilot_July 28, 2017 2
Innovative Assessment and Accountability • Allows for a pilot for up to seven (7) states to use competency-based or other innovative assessment approaches for use in making accountability determinations • Initial demonstration period of three (3) years with a two (2) year extension based on satisfactory report from the director of Institute for Education Sciences (IES), plus a potential 2 year waiver • Rigorous assessment, participation, and reporting requirements and subject to a peer review process • Maybe used with a subset of districts based on strict “ guardrails ,” with a plan to move statewide by end of extension Lyons & Marion_Comparability Options for the Innovative Pilot_July 28, 2017 3
Innovative Assessment and Accountability What does “innovative” mean? • Approved states may pilot with a subset of May Pilot in a Subset districts before scaling the system statewide of Districts by the end of the Demonstration Authority. • Approved states may design an assessment or Can Be Entirely system of assessments that consists of all Performance-Based performance tasks, portfolios, or extended learning tasks. • Approved states may assess students when they are ready to demonstrate mastery of Can Administer when standards and competencies as applicable so Students are Ready long as states can also report grade-level information. Lyons & Marion_Comparability Options for the Innovative Pilot_July 28, 2017 4
Purpose of ESEA “From the beginning, Title 1 of ESEA included assessment and accountability requirements as a safeguard to ensure that the federal money being allocated to programs to improve the achievement of the disadvantaged was being spent wisely.” (DePascale, 2015) – The purpose of ESEA accountability is to ensure that public tax dollars are resulting in improved educational programming and the intended student outcomes related to achievement and equity (Bailey & Mosher, 1968). Lyons & Marion_Comparability Options for the Innovative Pilot_July 28, 2017 Page 5 • Lyons • NCME 2017
Why Should We Care About Comparability? 1. Fairness: Because states must use assessment results from the pilot districts in the state accountability system. 2. Equity in Opportunity to Learn: Make sure that the pilot districts are not getting a “hall pass”, all students are held to same expectations. Lyons & Marion_Comparability Options for the Innovative Pilot_July 28, 2017 6
Too Narrow of a Focus on Comparability A narrow focus on pilot to non-pilot comparability misses the bigger picture in two important ways: – by failing to address additional, and potentially more important, comparability questions, and – by potentially inhibiting innovation . Lyons & Marion_Comparability Options for the Innovative Pilot_July 28, 2017 7
Building an Evidence-Base for Score Comparability Comparable The focus of Annual Determinations the regulations Common achievement level descriptors and common Non-pilot Pilot Results assessments in select Results grades/subjects. Social moderation comparability audits on common and local tasks, District A District B standard setting, and validating Results Results pilot performance standards with samples of student work. Scoring calibration sessions, external audits on inter-rater reliability, audits on the Within District Within District generalizability of the local Results Results scores, reviews of local assessment quality and alignment. Lyons & Marion_Comparability Options for the Innovative Pilot_July 28, 2017 8
Threat to Real Innovation Legitimate reasons for non-comparability: 1. To measure the state-defined learning targets more efficiently (e.g., reduced testing time); 2. To measure the learning targets more flexibly (e.g., when students are ready to demonstrate “mastery”); 3. To measure the learning targets more deeply ; or 4. To measure targets more completely (e.g., listening, speaking, extended research, scientific investigations). Lyons & Marion_Comparability Options for the Innovative Pilot_July 28, 2017 9
Threat to Real Innovation Legitimate reasons for non-comparability: 1. To measure the state-defined learning targets more efficiently (e.g., reduced testing time); 2. To measure the learning targets more flexibly (e.g., when students are ready to demonstrate “mastery”); 3. To measure the learning targets more deeply ; or 4. To measure targets more completely (e.g., listening, speaking, extended research, scientific investigations). “Perfect agreement would be an indication of failure.” – Dr. Robert Brennan Lyons & Marion_Comparability Options for the Innovative Pilot_July 28, 2017 10
Comparability by Design If comparability is not How does the design of achieved, how will the How will the state the innovative state adjust the evaluate the degree of assessment system yield classification scale to comparability achieved evidence to support account for systematic across differing comparability claims? differences across assessment conditions? assessment systems? Lyons & Marion_Comparability Options for the Innovative Pilot_July 28, 2017 11
Comparability by Design The focus of the regulations If comparability is not How does the design of achieved, how will the How will the state the innovative state adjust the evaluate the degree of assessment system yield classification scale to comparability achieved evidence to support account for systematic across differing comparability claims? differences across assessment conditions? assessment systems? Lyons & Marion_Comparability Options for the Innovative Pilot_July 28, 2017 12
What’s Our Inference? • Many comparability studies focus on item- and score-level interchangeability • The innovative pilot requires comparability at the level of the annual determination – In other words, would a student considered proficient in one district also be considered proficient in another district given the same level of work? Lyons & Marion_Comparability Options for the Innovative Pilot_July 28, 2017 13
Expanding our notions of comparability Adapted from Winter (2010) Lyons & Marion_Comparability Options for the Innovative Pilot_July 28, 2017 14
Two Major Categories of Evidence 1. The alignment of the assessment systems to the content standards We strongly recommend that evidence of alignment for the – two assessment systems should come from alignment to the content standards rather than alignment to one another. 2. The consistency of achievement classifications across the two systems. Lyons & Marion_Comparability Options for the Innovative Pilot_July 28, 2017 15
Two Major Categories of Evidence 1. The alignment of the assessment systems to the content standards We strongly recommend that evidence of alignment for the – two assessment systems should come from alignment to the content standards rather than alignment to one another. 2. The consistency of achievement classifications across the two systems. The focus of the regulations Lyons & Marion_Comparability Options for the Innovative Pilot_July 28, 2017 16
Comparability Options in the Regulations • Administering both the innovative and statewide Audit assessments to all students in pilot schools at least once in any grade span • Administering full assessments from both the innovative and statewide assessment system to a demographically Sample representative sample of students at least once every grade span • Including common items in both the statewide Common Items and innovative assessment system • This is where we come in. We Other needed to offer additional options! Lyons & Marion_Comparability Options for the Innovative Pilot_July 28, 2017 17
16 Design Options for Evaluating Pilot to Non-Pilot Comparability in Rigor of Performance Standards No Students in All Students Some Students Common Both Measures Some Measures Third Measure in Common Other Lyons & Marion_Comparability Options for the Innovative Pilot_July 28, 2017 18
16 Design Options for Evaluating Pilot to Non-Pilot Comparability in Rigor of Performance Standards No Students in All Students Some Students Common Concurrent (in past): “Pre-equating” Both Measures Concurrent: Embedded common Some Measures items across both systems Concurrent: Third Measure Common independent in Common assessment Other Lyons & Marion_Comparability Options for the Innovative Pilot_July 28, 2017 19
Recommend
More recommend
