Maximizing Instructionally-Relevant Information from Interim - PowerPoint PPT Presentation

Maximizing Instructionally-Relevant Information from Interim Assessments Scott Marion, Center for Assessment Symposium on Learning-Focused Balanced Assessment Systems Presented at CCSSO’s National Conference on Student Assessment June 28, 2017

Purpose and Use • To illustrate the challenges of generating instructionally-relevant information from interim assessments, I focus on two of Joseph’s purposes: – Evaluating achievement of content (& skills) – Instructional planning/correction 2 Marion. Balanced Assessment Session. CCSSO June 28, 2017

Assumptions and Hopes • We can’t just say (hope) that our assessment(s) can evaluate achievement and/or support instruction without a clear specification of how that can happen 3 Marion. Balanced Assessment Session. CCSSO June 28, 2017

Evaluating achievement of content & skills • Why do we want to evaluate achievement? – Grading/competency determinations? – Curricular/program evaluation? – Identify students for remediation/enrichment? – Identify students for course placement? • When/how often do we want evaluate achievement? – At the end of a unit of instruction? – In the middle of a unit of instruction? – At the end of term or year? • What do we want as the focus? – All learning targets in the unit/year? – Focus on high-priority targets in depth? – A balance of breadth and depth? 4 Marion. Balanced Assessment Session. CCSSO June 28, 2017

Evaluating achievement of content & skills • If we want to evaluate achievement – For grading/competency determinations – At the end of a major unit of instruction – That balances depth and breadth • What are key design requirements? – Timing of assessment and return of results? Always need – If the only assessment used, must meet certain to ask “Why reliability/generalizability levels and how?” – Meets specified alignment requirements – Uses items and tasks that embody the enacted curriculum 5 Marion. Balanced Assessment Session. CCSSO June 28, 2017

Can Interim Assessments Play a Role? • Well-designed curriculum-embedded assessments would be the ideal way to meet these intended uses and criteria, but what about interim assessments ? Two main designs • Mini-summative designs essentially replicate the summative assessment “blueprint” on each interim assessment • Modular assessment designs are tied to specific aspects of the full content standards, but each assessment focuses on just a limited subset of the full domain 6 Marion. Balanced Assessment Session. CCSSO June 28, 2017

Example 5 th Grade Math Interim Mini-Summative Design Would Not Meet Our Needs Summative Design • Operations & Mini-summative #1 Algebraic Thinking • Operations & Algebraic Thinking • Number-Base 10 • Number-Base 10 • Number-Fractions • Number-Fractions • Measurement & Data • Geometry • Measurement & Data • Geometry Yet, this form of most interim tests! Marion. Balanced Assessment Session. CCSSO June 28, 2017 7

Example 5 th Grade Math Interim Modular Design Summative Design Operations & Algebraic Thinking Module • Operations & • Write and interpret Algebraic Thinking numerical expressions. • Analyze patterns and • Number-Base 10 relationships. • Number-Fractions • Measurement & Data • Geometry Marion. Balanced Assessment Session. CCSSO June 28, 2017 8

In Search of Balanced Assessment Systems • Modular interim assessments could be procured as part of a summative assessment RFP – Coherent learning targets – Same item specifications – Same item formats • Interim assessments could be optional for districts tied to appropriate points in the curriculum • Also helps deal with the “subscore conundrum” on summative tests! 9 Marion. Balanced Assessment Session. CCSSO June 28, 2017

Could An Interim Assessment Meet the Need? • For grading/competency determinations? • At the end of a major unit of instruction? • That balances depth and breath? • Interims could meet the timing and alignment requirements… • Will it be tied closely enough to the enacted curriculum to be useful for grading? • What about a competency determination? • What else would you need to know? 10 Marion. Balanced Assessment Session. CCSSO June 28, 2017

Let’s revisit our potential uses and purposes Could an interim system, coherently designed with the state system, meet any of the following needs? • Grading/competency ? Maybe as a part? determinations? • Curricular/program  Likely yes evaluation? • Identify students for ? Potentially useful remediation/enrichment? source of data • Identify students for ? Potentially useful course placement? source of data Marion. Balanced Assessment Session. CCSSO June 28, 2017 11

Purpose 2 : Instructional planning and feedback • What are the characteristics of an assessment that can be useful for informing instruction (i.e., formative)? 12 Marion. Balanced Assessment Session. CCSSO June 28, 2017

Mechanisms and Processes • For an assessment to inform instruction we must consider: – Timing – Relationship to specific curriculum – Types of items/tasks (designed to provide summary information or insight into student learning) – Form of the results and feedback (summaries, descriptions) – Level of support 13 Marion. Balanced Assessment Session. CCSSO June 28, 2017

Timing • Interim assessments that are administered only several times each year cannot meet the timing criterion – Is this a deal breaker for having interim assessments serve instructional purposes? – Must avoid Shepard’s “1000 mini-lessons” phenomenon Student 1 Student 2 Student 3 Student 4 Student 5 Student 6 14 Marion. Balanced Assessment Session. CCSSO June 28, 2017

Must be tied to specific curriculum • Formative assessments must “embody” Can modular learning goals interim assessments • Formative assessments and processes meet this must be curriculum-embedded criterion? – Tasks are instructional tasks so no instructional time is lost; occurs “midstream” to inform instruction not as a unit summative test • The curriculum provides the learning targets and, importantly, provides a framework for interpretation and next steps 15 Marion. Balanced Assessment Session. CCSSO June 28, 2017

Items and Tasks • Formative processes rely on a broad range of items, questions, observations, and extended tasks • Most commercial interim assessment systems rely almost exclusively on OR multiple-choice or other selected response formats – Could a modular interim include more than selected-response items? Perhaps? But will that be enough? 16 Marion. Balanced Assessment Session. CCSSO June 28, 2017

Nature of Results/Reports/Feedback • Interim assessment results are aggregated beyond single classrooms – Involves some type of data reduction – Growing body of work about effective score reporting can be utilized to design reports as instructionally useful as possible – Types of available data summaries can provide useful evaluative information, but not necessarily instructional • For interim assessments to provide instructionally useful information, qualitative information must be reported – Information on correct/incorrect responses by sub-domain – Feedback on what an incorrect answer implies – Suggestions for next steps 17 Marion. Balanced Assessment Session. CCSSO June 28, 2017

Support for improvement • If teachers could do this already, we wouldn’t be here • We must help support teachers as they take data and information and turn it into decisions and actions – Models – Mentoring – On-going training • Teachers receive these types of supports when they engage in high- quality professional development to learn formative assessment processes 18 Marion. Balanced Assessment Session. CCSSO June 28, 2017

Checking In • Can modular interim assessments: – Evaluate achievement of content (& skills)? Maybe – Support instructional planning/correction? Doubtful • Some assessments may help inform and improve teaching and learning without meeting all the criteria for formative assessment, but certain features may be critical: – Providing qualitative insights about understandings and misconceptions not just a numeric score – Providing rich models of instructional activities – Giving timely feedback on what to do besides re-teaching every missed item • It is hard to imagine how an assessment administered only 3-4 times during the year can be “formative” 19 Marion. Balanced Assessment Session. CCSSO June 28, 2017

Questions, comments or more information: Center for Assessment www.nciea.org smarion@nciea.org