Rethinking our Alignment Obsession: Can a Generalizability Perspective Help? Scott Marion, Center for Assessment CCSSO National Conference on Student Assessment Aligned to What: Complex Content Standards as Targets for Assessment Design and Alignment Evaluation June 29, 2018
Early Days of Standards-Based Reform Alignment was the promissory note: “We’ll assess only the Content stuff you’re expected to Standards teach.” Performance Standards Let’s not forget “School Assessments Delivery” or “Opportunity to Learn” standards Marion. Generalizability approach to alignment_June 29, 2018 2
Alignment Made Sense in the Early Days • We were moving out of an era of norm- referenced tests • Theory of action behind standards-based systems: – Make content and performance standards explicit – Base assessments on those standards – Fair to educators and would lead to improved learning Marion. Generalizability approach to alignment_June 29, 2018 3
From Proxy to Criterion • Alignment started off a proxy for content-related validity evidence • Once certain methodologies (e.g., Webb) became popularized and used in USED standards and assessment reviews, alignment became reified as the criterion • While embracing the fairness concerns that alignment was intended to address, we now are experiencing some important unintended negative consequences? – Negative or positive might depend on your view of curriculum and assessment Marion. Generalizability approach to alignment_June 29, 2018 4
Complex Content Standards Increasing the complexity of the domain representation found in the Crosscutting newer standards (e.g., CCSS, NGSS) Core Concepts Ideas leads to worries about fairly and Practices appropriately representing the curriculum (standards) targets on the assessment Marion. Generalizability approach to alignment_June 29, 2018 5
Sparse Content/Curriculum Coverage My colleague Brian Gong has been concerned with various aspects of construct underrepresentation in the NGSS. This shows the sparse coverage of the Performance Expectations (PE) in middle school life science. Remember PEs are not all possible intersections of the three dimensions. Science and Engineering Practices PE x DCI Middle School AQDP DUM PCOI AID UMCT CEDS EAE OECI Totals LS1.A, Structure and MS-LS1-2 MS-LS1-1 MS-LS1-3 3 Function LS1.B, Growth and MS-LS1-5 MS-LS1-4 2 8 Development of Organisms LS1, From Molecules to Organisms: Structures and LS1.C, Organization for MS-LS1-7 MS-LS1-6 Processes Matter and Energy Flow in 2 Organisms LS1.D, Social Interactions MS-LS1-8 1 and Group Behavior MS-LS2-1 MS-LS2-2 LS2.A, Interdependent 2 Relationships in Ecosystems LS2.B, Cycles of Matter and MS-LS2-3 Life Sciences Energy Transfer in LS2, Ecosystems: 1 5 Ecosystems Interactions, Energy, and LS2.C, Ecosystem MS-LS2-4 Dynamics Dynamics, Functioning, and MS-LS2-5 2 Resilience LS2.D, Social Interactions 0 and Group Behavior MS-LS3-1 LS3.A, Inheritance of Traits MS-LS3-2 2 2(2) LS3, Heredity: Inheritance MS-LS3-1 and Variation of Traits LS3.B, Variation of Traits MS-LS3-2 2 MS-LS4-1 LS4.A, Evidence of MS-LS4-2 MS-LS4-3 3 Common Ancestry MS-LS4-4 MS-LS4-5 2 6 LS4, Biological Evolution: LS4.B, Natural Selection MS-LS4-6 Unity and Diversity 1 LS4.C, Adaptation LS4.D, Biodiversity and 0 Humans PE x SEP Totals 0 5 1 3 1 5 4 2 21 From Brian Gong (2018). Paper under development. www.nciea.org. Marion. Generalizability approach to alignment_June 29, 2018 6
Sparse Assessment Coverage • Even if the state has adopted the PEs as the “standards,” there is still way too much to cover well on a 2-3 hour exam • If you are assessing only at grade spans, then the problem is even worse Finally, I know many are enamored with item clusters, but I question the fidelity to the Frameworks and NGSS Marion. Generalizability approach to alignment_June 29, 2018 7
What Do We Care About? • Kids doing real “minds on” science • Posing questions about phenomena, designing studies, & reasoning with evidence in light of big ideas of science Marion. Generalizability approach to alignment_June 29, 2018 8
From Alignment to Generalizability • But what about the crushing weight of alignment requirements? • Would a generalizability framework be more useful? Marion. Generalizability approach to alignment_June 29, 2018 9
Generalizability and Transfer • Generalizability is the measurement analogue of learning transfer • This requires us to be clear about the target of generalization and the factors that would influence or threaten such generalizations – See TJ’s “Jill and Jack” – Yes, this is hard! • We need to focus on providing credible evidence that the student really knows what is being claimed the student knows based on the assessment scores. Marion. Generalizability approach to alignment_June 29, 2018 10
Generalizability and Validity • Both alignment and generalizability are key components of a validity argument , but generalizability applies more directly especially when using Kane’s argument-based approach *Thanks to Brian Gong for the diagram Focus of argument Marion. Generalizability approach to alignment_June 29, 2018 11
So What Could This Look Like? • Good —a set of rich performance tasks to be administered over a defined period of time or a set of performance assessments supplemented by more easily administered and scored items • Better —as part of a full system of assessments • Best —as part of a full system of assessments with tasks embedded as part of defined curricular units Marion. Generalizability approach to alignment_June 29, 2018 12
Typical Approach to Assessment Systems Controlling the This is assuming we conversation have a system! Large Scale Assessments District required common assessments Classroom and/or school based assessment Marion. Generalizability approach to alignment_June 29, 2018 13
What If We Flipped the System? Evidence of student learning from classroom assessments and other artifacts State and District Required Embedded Assessments Focus of Large scale generalizability performance analyses assessments Marion. Generalizability approach to alignment_June 29, 2018 14
Yeah, yeah… • I know all too well about the challenges with reliability, field testing, equating, comparability, etc., but I contend that in chasing alignment, we’re running the risk of sacrificing rich science learning … • Misplaced concern about student-level reliability – Few, if any, statewide science assessments are used for student-level decisions—rather, they are used at the school- level if at all – Hill & DePascale (2003) convincingly demonstrated that we can achieve respectable school-level reliabilities (e.g., .90+) even with student-level reliabilities around .65 (assuming approx. 50 students in the school) Marion. Generalizability approach to alignment_June 29, 2018 15
Focusing on the Real Thing! The business of schools is to invent tasks, activities, and assignments that the students find engaging and that bring them into profound interactions with content and processes they will need to master to be judged well educated . Schlechty, P.C. (2001) Shaking up the schoolhouse . San Francisco: Jossey-Bass. Marion & Shepard, iNACOL Leadership Webinar June 15, 2017 16
For more information: Center for Assessment www.nciea.org smarion@nciea.org Marion. Generalizability approach to alignment_June 29, 2018 17
Recommend
More recommend
