E ng ine e ring PL Ds and T e st Co nte nt to E ng ine e r Cut - PowerPoint PPT Presentation

E ng ine e ring PL Ds and T e st Co nte nt to E ng ine e r Cut Sc o re s OR Aligning Test Development with Intended Score Interpretations Steve Ferrara A presentation in Engineered Cut Scores: Aligning Standard Setting Methodology with Contemporary Assessment Design Principles , a session in the National Conference on Student Assessment June 22, 2016

Ove rvie w Define and illustrate test content ‐ PLD alignment  Principled design and validity argumentation call for this  alignment Engineering PLDs  Engineering test forms  Engineering items  Concept not new; engineering PLDs, test forms, and items is WIP  Engineered Cut Scores NCSA Session 2

Response demands: Content, Response demands: Content, cognitive, and linguistic features of cognitive, and linguistic features of Pre mise items that are related to item items that are related to item difficulty and quality (i.e., difficulty and quality (i.e., discrimination) discrimination)  Need a broader conception of alignment to engineer cut scores  Proposed: Alignment is the degree to which an item’s response demands are consistent with the knowledge and skill requirements described in the corresponding PLD  Aside: Need a broader type of alignment to provide evidence to support intended score interpretations and uses, as well Engineered Cut Scores NCSA Session 3

What is te st c o nte nt-PL D alig nme nt?  High Level 4 xxx Item Response Demands Aligned Aligned Meets xxx Content, Cognitive, Linguistic Expectations xxx Consistent with Level 4 KSAs Level 3 xxx Item Response Demands Approaches xxx Content, Cognitive, Linguistic Expectations xxx Consistent with Level 3 KSAs Level 2 xxx Item Response Demands Partially Meets xxx Content, Cognitive, Linguistic Articulated Articulated Expectations xxx Consistent with Level 2 KSAs  Low Level 4: Approaches Expectations for assessed content Very complex text: General accuracy and understanding Moderately complex text: General accuracy and understanding PARCC PLDs, Readily accessible text: Mostly accurate analyses, showing understanding grade 6 Reading; http://www.parccon line.org/assessment Level 3: Approaches Expectations for assessed content s/test ‐ design/ela ‐ Very complex text: Minimal accuracy and understanding literacy/ela ‐ Moderately complex text: General accuracy, basic understanding performance ‐ level ‐ Readily accessible text: Mostly accurate analyses, showing understanding descriptors Level 2: Partially Meets Expectations for assessed standards Very complex text: Inaccurate analysis, limited understanding Moderately complex text: Minimal accuracy and understanding Readily accessible text: Partial accuracy and understanding Engineered Cut Scores NCSA Session 4

Alignment Alignment within and within and across grades across grades T e st c o nte nt-PL D alig nme nt and artic ulatio n no t auto matic Misalignment Misalignment Misalignment Misalignment Misalignment Misalignment Misalignment Misalignment Ferrara, Svetina, Skucha, & Davidson, 2011 Engineered Cut Scores NCSA Session 5

L inking re ading c o mpre he nsio n ite ms to the Co mmo n E uro pe an F rame o f Re fe re nc e (CE F R) Figueras, N., Kaftandjieva, F., & Takala, S. (2013) Engineered Cut Scores NCSA Session 6

PRINC IPL ED A PPRO A C HES Engineered Cut Scores NCSA Session 7

Princ iple d appro ac he s to asse ssme nt de sig n, de ve lo pme nt, and imple me ntatio n  Several names, several conceptualizations  Common elements, varying details (Ferrara, Lai, Reilly, & Nichols, 2016) ◦ Evidence Centered Design (ECD) ◦ Assessment Engineering (AE) ◦ Cognitive Design Systems (CDS) ◦ BEAR Assessment System ◦ Principled Design for Efficacy (PDE) Engineered Cut Scores NCSA Session

Princ iple d appro ac he s: De sig n (e tc .) fo r alig nme nt Table 1 Foundation and Organizing Elements of Principled Approaches to Assessment Design, Development, and Implementation and their Relationship to the Assessment Triangle Framework Elements Assessment Triangle Alignment Organizing Element Ongoing accumulation of evidence to support validity arguments Overall evidentiary reasoning goal Intended score Intended score interpretations interpretations Foundational Elements Clearly defined assessment targets Cognition Statement of intended score interpretations and uses Cognition Model of cognition, learning, or performance Cognition Aligned measurement models and reporting scales Interpretation Alignment Alignment Manipulation of assessment activities to align with assessment targets Observation and intended score interpretations and uses From Ferrara, Lai, Reilly, & Nichols (2017) Principled approaches are Assessment Engineering, BEAR Assessment System, Cognitive Design System, Evidence Centered Design, and Principled Design for Efficacy Engineered Cut Scores NCSA Session 9

ENG INEERING PL DS Engineered Cut Scores NCSA Session 10

L o g ic o f infe re nc e s PLDs Everything Ferrara, Lai, Reilly, & Nichols, 2016, Fig. 1 else, including items, aligned to PLDs Engineered Cut Scores NCSA Session

F rame wo rk fo r de ve lo ping PL Ds From Egan, Schneider, & Ferrara, 2012, Table 2 Engineered Cut Scores NCSA Session

Guidanc e o n e ng ine e ring PL Ds  Develop PLDs to guide test development, articulate standards across grades, link to learning trajectories (Bejar, Braun, & Tannenbaum, 2006, 2007)  Policy definitions ◦ Generic, by policy makers (Loomis & Bourque, 2001; Perie, 2008) ◦ Number of levels; labels and meaning (Beck, 2003; Burt & Stapleton, 2010; Cizek & Bunch, 2007; Egan, Schneider, & Ferrara, 2012; Perie, 2008; Zieky, Perie, & Livingston, 2008)  Range and other PLDs ◦ Explicit about content knowledge (Egan et al., 2012; Mills & Jaeger, 1998; Perie, 2008; US Department of Education 2004) ◦ Explicit about cognitive processes (Egan et al., 2012; Perie, 2008) ◦ Nouns and verbs, defining phrases (Egan et al., 2012)  Would like to find guidance on working from policy PLDs to content standards to range PLDs Engineered Cut Scores NCSA Session 13

PURSUING PL D- IT EM A L IG NMENT Engineered Cut Scores NCSA Session 14

Ho w c an we pursue alig nme nt?  Code items for response demands  Determine which items are aligned with the KSA requirements in the corresponding PLD  Build aligned test forms WIP: Next WIP: Next study, I study, I ◦ Select only aligned items hope hope  Item development ◦ Develop item specifications and item writer training to improve alignment ◦ Re ‐ field test items that are not aligned to PLDs Engineered Cut Scores NCSA Session 15

ENG INEERING T EST FO RMS FO R A L IG NMENT : IL L UST RA T IO N Engineered Cut Scores NCSA Session 16

Se le c te d ite m re spo nse de mand c o de s  Question Type ◦ The cognitive task an item poses (e.g., explain, analyze)  Depth of Knowledge ◦ Recall, Skill/concept, Strategic thinking  Relational Complexity ◦ No. of facts, concepts, and skills to be processed  Linguistic Complexity ◦ No. of prepositional phrases, as a proxy  Command of Textual Evidence ◦ Single, multiple pieces of text  Response Mode ◦ Select response, multiple responses, construct responses Engineered Cut Scores NCSA Session 17

E xc e rpt fro m Of F at, F e athe rs…; ite m 3874 Finger here Finger Finger here here Engineered Cut Scores NCSA Session

Item 3874 PLD Alignment Target Moderate Complexity text: Minimal accuracy and understanding Do correct responses to this item support this intended interpretation (or claim)? Item Response Demands Question Type USE DOK level 2. Skill/concept Relational 5 Complexity Number of 5 Prepositions Command of Low Textual Evidence Response Mode Low Engineered Cut Scores NCSA Session 19

E xc e rpt fro m T urn, T urn My Whe e l and ite m 3182 Prior knowledge not going to help

Item 3182 PLD Alignment Target Readily Accessible text: Partial accuracy and understanding Do correct responses to this item support this intended interpretation (or claim)? Item Response Demands Item Response Demands Question Type USE and INF Question Type USE DOK level 3. Strat. Thinking DOK level 2. Skill/concept Relational 5 Relational 6 or more Complexity Complexity Number of 5 Number of 7 Prepositions Prepositions Command of Low Command of Low ‐ Moderate Textual Evidence Textual Evidence Response Mode Low Response Mode Low Engineered Cut Scores NCSA Session 21

E xc e rpt fro m A S ing le S hard: ite m 3175 “Arid” seems pretty clear; “cold,”, “long,” and “rotten” seem unlikely. So what makes this relatively difficult? Prior knowledge of “arid” might help, but “arid bones” is figurative language. Engineered Cut Scores NCSA Session 22

Item 3175 PLD Alignment Target Very complex text: Inaccurate analysis, limited understanding Do correct responses to this item support this intended interpretation (or claim)? Item Response Demands Item Response Demands Question Type USE and INF Question Type USE or INF DOK level 3. Strat. Thinking DOK level 2. Skill/concept Relational 6 or more Relational 5 or more Complexity Complexity Number of 7 Number of 4 Prepositions Prepositions Command of Low ‐ Moderate Command of Low Textual Evidence Textual Evidence Response Mode Low Response Mode Low Engineered Cut Scores NCSA Session 23

ENG INEERING IT EMS FO R A L IG NMENT Engineered Cut Scores NCSA Session 24