A Data-informed Judgment-based Procedure For Linking Cut Scores On - PowerPoint PPT Presentation

A Data-informed Judgment-based Procedure For Linking Cut Scores On Alternative Assessment Formats: BACKGROUND Phoebe Winter National Conference on Student Assessment, June 27, 2018

Test Development for ELPA21 ■ Evidence-Centered Design Approach – Claims and Subclaims -> Evidence -> Tasks – Universal Design principles ■ Detailed item/task specifications – Measurement target – Level(s) addressed – Task structure, parameters around stimulus, stems, responses – Scoring rules/rubrics – Sample task(s) ■ Item/task content and bias review ■ Field test ■ Item/task selection for operational pool 2

Braille Version ■ Guiding principle: braille version measures same language skills ■ Expert review of online items for accessibility – Accessible with different presentation/response mode – Accessible with minor changes – Substitute item/task needed that measures same standard, e.g., ■ Different speaking prompt ■ Manipulatives ■ Content and bias reviews of braille form items ■ Operational braille item pool 3

Relationships between Braille and Online Items/Tasks Group Relationship 1 The braille version is identical to the online item, other than necessary differences in the format of presentation; changes are superfluous to apprehending the meaning of the item. 2 Similar to level 1, but the braille form version response format is modified or a visual that orients students to the text has been deleted. For example, pictures that accompany a read-aloud story are dropped. 3 While these braille form items are based on the online item, revisions cause the depth of concept measured to be different from the online versions. For example, a graph in the online version is changed to a table. 4 These braille form items are designed to measure a concept from the ELPA21 standards and ALDs, but the items have different content from the online version. For example, an online item may ask students to compare two pictures, while the modified item measuring the same standards asks the student to compare two events. 4

Example of Group 1: Very Similar Narrator: Read about animals. Then answer the questions. Passage: Animals must protect themselves from danger. Some animals change colors. Some animals have sharp teeth. Some animals have hard shells on the outside of their bodies. When a turtle sees danger, it pull its body into its shell. An armadillo is different. When it is scared, it rolls into a ball. The hard plates on its back make a shield so other animals cannot hurt it. 5

Example of Group 2 or 3 Narrator: Listen and answer the question. Hannah likes to do her homework in the morning before she goes to school. Tyler likes to do his homework right after school in the afternoon. Mia likes to do homework at night before going to bed. When do you think is the best time to do homework? Give three reasons to explain your opinion. 6

Example of Group 4: Substitute Item Needed Narrator and text : The four pictures below show a story about something that happened. What story do the pictures tell? Look at the pictures and prepare to write a paragraph. You may use the words in the Word Bank to help you. Type your story in the text box and click submit when you are finished. Be sure to check your work. Make sure you tell a complete story from beginning to end and organize ideas in a logical way. Narrator : Write a story about XXX. You may use the words in the Word Bank to help you. Be sure to check your work. Make sure you tell a complete story from beginning to end and organize ideas in a logical way. Word Bank: Word Bank: empty, litter; garden; clean; plants 7

The Dilemma ■ ELPA21 braille form measures language development in the same way and in the same contexts, as the online form ■ Students taking the braille form are asked to demonstrate the same skills as students taking the online form ■ The braille form addresses the ELP standards BUT ■ How do we report braille form results that are comparable to online form results? 8

Challenges to Deriving Comparable Scores ■ Common item linking – Small number of braille examinees – Precludes statistical linking, even assuming “direct braille” has same parameters as online ■ Common student linking – Braille form and online form cannot be accessed by the same students ■ Applying online version parameters – Many of the tasks are different on the braille form, either in presentation mode or in content or both – We cannot assume that even tasks presented in braille with no content changes have the same item parameters as online items 9

Approach ■ Judgment-based, data-informed linking process – Social moderation – Borrows from standard-setting procedures ■ Experts compare two versions of an item (when available), online and braille ■ Experts are given the expected performance of online students just at the cut score when available, along with a description of the skills that define performance that just meets the cut score ■ The task: determine the expected performance on the braille version for students who have the same level of targeted language skills as students at the online cut score ■ The result: for each item, judgments of the expected performance of students taking the braille item that is comparable, in terms of language skills, to the expected performance of students taking the online version 10