Empirical Methods for Evaluating Maps: Illustrations and Results W. - PowerPoint PPT Presentation

Empirical Methods for Evaluating Maps: Illustrations and Results W. Jake Thompson & Brooke Nash

Methods for Evaluating Map Structure • External outcomes • Classical item statistics • Unidimensional models 2

A Framework for Map Evaluation • Diagnostic Classification Models (DCMs) • Mastery profiles on the set of assessed skills • Three methods – Patterns of Mastery Profiles – Patterns of Mastery Assignment – Patterns of Attribute Difficulty 3

An Illustrative Example Initial Precursor Target • 3 attribute assessment • Linear map structure 4

Map Structure in a DCM Context 5

1 0 0 0 0 1 1 1 1 0 0 1 0 1 1 1 1 1 0 Patterns of Mastery Profiles hypothesized profiles 1 0 0 0 Target Precursor Initial 0 profiles checks • Estimate two models – Saturated model with all – Reduced model with only • Assess model fit – Posterior predictive model – Model comparisons 6

… 0 Target Precursor Initial Student … … … … 0 1 0 5 1 0 1 4 1 1 1 1 3 0 … … … 0 0 0 5 1 1 0 4 1 1 1 3 0 0 1 2 1 0 Patterns of Attribute Mastery 2 .83 .89 .92 3 .13 .52 .86 .43 0 .85 .97 1 Target Precursor Initial Student (equivalent to LCA) 4 .88 .65 .85 1 2 0 1 1 1 Target Precursor Initial Student … … … … .33 .70 .55 5 • Estimate each attribute as a separate 1-attribute DCM • Set mastery threshold (0.8) 7

classical p -values value for each attribute and group Patterns of Attribute Difficulty • Measure attribute difficulty using • Group similar respondents a priori • Calculate the weighted average p- 8

Case Study: Dynamic Learning Maps levels of depth, breadth, and complexity progression during the operational assessment • Each Essential Element (EE) available at multiple – 5 levels in ELA and mathematics – 3 levels in science • Linkage levels are assumed to follow a linear • Students test on only one linkage level for each EE 9

ELA missing data level reversals than lower Case Study: Dynamic Learning Maps • Patterns of Profile Mastery – Models fail to converge due to • Patterns of Attribute Mastery – The majority of flags were in – More flags for higher linkage 10

Case Study: Dynamic Learning Maps • Patterns of Attribute Difficulty – Flags by subject • 28 ELA EEs • 35 mathematics EEs • 0 science EEs 11

Summary framework information applied to future test and map development • Benefits and limitations of each method within the • Wide breadth of methods provides complementary • Application to DLM shows insights that can be 12

Ongoing Research Mastery Profiles criteria within EEs to the more fine-grained map structure • Continue to refine methods – Alternative modeling strategies for Patterns of – Simulation studies to inform empirical flagging • Expanding beyond the progression of linkage levels 13

More Information