Smarter Balanced Achievement Level Setting Dallas, TX, October - PowerPoint PPT Presentation

Smarter Balanced Achievement Level Setting Dallas, TX, October 2014

Developing the Expertise to Make Content-Based Achievement Level Recommendations • Day 1 Activities that must be completed before panelists are qualified to set achievement levels using the Bookmark Procedure: – Study content standards – Study achievement level descriptors – Understand what the tests measure • Take practice test • Study ordered item booklets using item maps

Ordered Item Books (OIBs) 22 21 • 20 Virtual OIB (online) 19 • 18 One item per page 17 • 16 Easiest item first 15 • 14 Items ascend by difficulty 13 • 12 Hardest item last 11 10 9 8 7 6 5 4 3 2 1 Ordered Item Book 3

4

5

6

Study the Items in the OIB As you study each item in the OIB, discuss two questions with your fellow panelists: 22 21 20 19 1. What do you know about a student who responds 18 17 successfully to this item; that is, what skills must a 16 15 student have in order to know the correct answer? 14 13 12 11 10 2. What makes this item more 9 difficult than preceding items? 8 7 6 5 This activity helps you 4 3 acquire the 2 knowledge to make 1 content-based cut Ordered score Item recommendations. Book

Study the Items in the OIB As you study each item in the OIB, discuss two questions with your fellow panelists: 22 21 20 19 1. What do you know about a student who responds 18 17 successfully to this item; that is, what skills must a 16 15 student have in order to know the correct answer? 14 . 13 12 11 10 2. What makes this item more 9 difficult than preceding items? 8 7 6 5 4 3 2 1 Ordered Item Book

9

Passages and Other Materials Item deleted to maintain test security

Grade Content Area Room Set Up Research Content Facilitator Facilitator One Table Facilitator at each table Tables of panelists and Table Leader

The Bookmark Procedure Smarter Balanced Achievement Level Setting Dallas, TX, October 2014

Bookmark Overview • The Bookmark Procedure is a process used to set cut scores that define achievement levels by comparing and aligning items and content to achievement level descriptors • It is so named because you place bookmarks in an ordered item booklet of test items to make your cut score recommendations

Threshold Student • We recommend cut scores for Threshold Students — the students with a level of achievement that just barely qualifies them to be in the achievement level – These are the skills of the student just entering the achievement level Threshold Level 3 Mid-level Level 3 High-achieving Level 3 Student Student Student Level 3 Level 4 Cut Score Cut Score 17

Three Threshold Students Level 4 Threshold Level 2 Threshold Level 3 Threshold Students Students Students Level 2 Level 3 Level 4 Cut Score Cut Score Cut Score 18

Three Cut Scores Define Four Achievement Levels Level 3 Level 4 Threshold Level 2 Threshold Threshold Students Students Students Level 4 Level 1 Level 2 Level 3 Students Students Students Students Level 2 Level 3 Level 4 Cut Score Cut Score Cut Score 19

Locating a Chess Player on a Scale 95% 82% 64% 50% 50% 32% 24% 11% 1% 71% chance chance chance chance chance chance chance chance chance chance Lower ability Higher Ability 1 2 4 5 6 7 8 9 3 20

Locating the Threshold Student on the Scale 95% 82% 74% 61% 50% 50% 43% 34% 21% 14% chance chance chance chance chance chance chance chance chance chance Easier items Harder Items 1 4 6 1 7 2 3 5 8 9 21

Locating the Threshold Student on the Scale >50% >50% >50% >50% 50% 50% <50% <50% <50% <50% <50% chance chance chance chance chance chance chance chance chance chance chance Easier items Harder Items 1 4 6 1 7 2 3 5 8 9 10 22

Bookmark Placement Instructions • Ask yourself: Would a student at the threshold have at least a 50% chance of earning this point? – Yes: Move on to the next item. – No: Place your bookmark here.

Items 6 – 22 reflect content for 22 which the threshold Level 3 student 21 would have less than a 50% chance 20 19 of success. 18 17 16 15 Items 1-5 reflect content for 14 which the Level 3 threshold 13 12 student would have at least 11 a 50% likelihood of success. 10 (with about a 50% chance of 9 success on the item just 8 7 before the bookmark ) L3 6 5 Bookmark on Page 6 4 3 2 1 Ask yourself: Would a student at the threshold have at least a 50% chance of Ordered It may be easiest for you to earning this point? Item first identify a range of Book Yes: Move on to the next item. items No: Place your bookmark here. 24

Level 3 Bookmark Placement • When you place your Level 3 bookmark, think about a student at the threshold of Level 3 based on the achievement level descriptors – Place your Level 3 bookmark at the point in the OIB such that – A threshold Level 3 student would be have at least a 50% likelihood of success on each of the items before the bookmark (and about a 50% chance of success on the item just before the bookmark) – A threshold Level 3 student would have less than a 50% likelihood of success on the items from the bookmark on – It may be easiest for you to first identify a range of items that begins where threshold Level 3 students begin to be challenged and end where they would definitely be challenged. Then pick the best page in that range. 25

Items 18 – 22: 22 less than a 50% chance of success. 21 20 L3 19 18 17 16 Items 1-17: 15 At least 50% chance of 14 success 13 12 11 10 9 8 7 6 5 4 3 Ask yourself: Would a student at the 2 threshold have at least a 50% chance of 1 earning this point? Ordered Yes: Move on to the next item. Item No: Place your bookmark here. No: Place your bookmark here. Book Bookmark on Page 18 26

22 21 20 L4 19 18 17 16 15 14 13 12 11 L3 10 9 8 7 6 L2 Three bookmarks 5 4 3 define four 2 1 achievement levels. Ordered Item Book 27

Setting Bookmarks 28

Agenda Day 1 • Morning: Study CCSS and ALDs • Afternoon: Study the Ordered Item Booklet Day 2 • Morning: Bookmark training and Round 1 – Make bookmark recommendations individually • Afternoon: Round 2 – Discuss differences in Round 1 bookmark placements at each table – Place Round 2 bookmarks individually Day 3 • Morning: Round 3 – Discuss Round 2 bookmark placements for the entire room (all panelists at all tables) – View supporting data based on Round 2 bookmarks (e.g., impact data, online panel results) – Place Round 3 bookmarks individually – Review final recommendations – Evaluate the process 29