Reports for supporting full-cycle assessment Rebecca Freund and Amy - PowerPoint PPT Presentation

Reports for supporting full-cycle assessment Rebecca Freund and Amy Arneson University of California at Berkeley

Overview

NRC ASSESSMENT TRIANGLE Observation Cognition Reports Interpretation 3

PURPOSES OF REPORTS • Informational – Present results in an accessible way • Interpretable – Help users understand the information • Educational – Guide a deeper understanding of a concept 4

ASSESSMENT DEVELOPMENT CYCLE Four building blocks Construct Definition Measurement Item Design Model Outcome Space Definition 5

ASSESSMENT DEVELOPMENT CYCLE Creation of scoring guides Construct Definition Measurement Item Design Model Outcome Space BASS scoring collection begins Definition 6

ASSESSMENT DEVELOPMENT CYCLE Reports Construct Post-Psychometric User facing Definition Analysis Supporting Researcher Measurement Psychometric facing Item Design Model Analysis Pre-Psychometric User facing Outcome Space Analysis Definition 7

Pre-analysis reports

MOTIVATION • Provides useful information to teachers • Encourages continued use of the application • Supports conceptual understanding to prepare for later, more complex reports 9

TYPES OF REPORTS • Group report – How many students responded at each level? • Individual report – How did this student respond to each item? 10

GROUP REPORT • Size of bars = proportion of students at each level of item 11

GROUP REPORT • Size of bars = proportion of students at each level of item • Hover to see level and student names 12

GROUP REPORT • Size of bars = proportion of students at each level of item • Hover to see level and student names • Organized by construct 13 •

INDIVIDUAL REPORT • Key showing all levels of construct 14

INDIVIDUAL REPORT • Key showing all levels of construct • Shapes show available levels for each item 15

INDIVIDUAL REPORT • Key showing all levels of construct • Shapes show available levels for each item • Colors show location of that student’s response 16

Psychometric reports

PRINCIPLES • Show persons and items on the same scale • Check for coverage at different levels of ability • Identify items that show misfit 18

GROUP REPORT: WRIGHT MAP • Left panel shows location of students • Right panel shows location of item levels • Identify difficulty bands and outlier items and levels 19

INDIVIDUAL REPORT: KIDMAP • Horizontal line shows student location • Left side shows levels reached (scored at at least that level), right side not reached • Yellow regions are unexpected 20

Post-analysis reports

GOALS • Show overall performance in a construct • Present student results in terms of psychometrics in accessible ways • Bring together all the information in the system 22

INDIVIDUAL REPORT • Width of levels suggests ability increase needed to advance 23

INDIVIDUAL REPORT • Width of levels suggests ability increase needed to advance • Student location in terms of construct levels, with error bars 24

INDIVIDUAL REPORT • Width of levels suggests ability increase needed to advance • Student location in terms of construct levels • Hover to see names of levels 25

GROUP REPORT • Location of all students in terms of construct levels • Assess the proficiency level of the class as a whole • Identify and compare individual students 26

OVERALL REPORT PRINCIPLES • Informational – How students performed on the items and in the construct • Interpretable – Connecting responses to levels of the construct • Educational – Strengthen teachers’ understanding of performance in terms of construct levels, vs. right/wrong or arbitrary number of points 27

Thank you!