Improving Technology Enhanced Items Eric E. Zilbert National - PowerPoint PPT Presentation

Improving Technology Enhanced Items Eric E. Zilbert National Conference on Student Assessment June 24–26, 2019 Orlando, Florida

Presentation Overview •What we know about Technology Enhanced Items (TEI). •What we at the California Department of Education (CDE) have done to try and improve TEIs. •What we need to think about to continue to improve TEIs.

What we know •There are many types. •Most tend to be harder than single select multiple choice items. •Item type can impact test delivery, scoring, and standard setting. •Item type can impact use of accessibility features. •Little information is provided to item writers about when to select what type of item and why.

Many types of Items • Overall CDE has identified 16 possible item types for use on the science assessments and alternate assessments. • SBAC uses some additional item types, e.g. text select.

1. Multiple Choice, Single Select MCSS 2. Multiple Choice, Multiple Select MCMS 3. Zone, Single Select ZoneSS 4. Zone, Multiple Select ZoneMS 5. Grid Multiple Select GridMS 6. Inline Choice List Single Select InlineChoiceListSS 7. Inline Choice List Multiple Select InlineChoiceListMS 8. Match, Single Select MatchSS 9. Match, Multiple Select MatchMS

More item types … 10. Extended Text 11. Fill-in-blank 12. Multiple Fill-in-blank 13. Composite 14. Composite CR 15. Interactive 16. Interactive CR

Subject & Score N P-value Grade Item Type Points EL3 TextExtractionMS - Member 0,1 4,839 0.03 EL6 FillInBlank - Discrete 0,1,2* 825 0.34 EL7 MCMS - Member 0,1 569 0.04 E11 Composite - Member 0,1 2,742 0.04 MA5 MatchMS - Discrete 0,1 507 0.04 MA5 GridMS - Discrete 0,1 919 0.03 MA5 ExtendedText - Discrete 0,1 525 0.04 MA7 TextEntry - Discrete 0,1 3,425 0.04 MA7 FillInBlank - Member 0,1 925 0.03 MA8 TextEntry - Discrete 0,1 919 0.02 MA8 TextEntry - Discrete 0,1 999 0.02 M11 TextEntry - Member 0,1 4,138 0.04 M11 ExtendedText - Discrete 0,1 3,829 0.03 M11 TextEntry - Discrete 0,1 751 0.04 M11 MCMS - Discrete 0,1 530 0.04

Of 41 flagged items • 11 multiple choice multiple select • 8 text extraction multiple select • 6 fill in blank (1 too easy) • 5 text entry items • 3 extended text • 3 grid multiple select • 3 composite • 1 match multiple select • 1 multiple choice single select (too easy)

Purpose 1. Describe the item types that will be used on the CAST; 2. Define the features of each item type and the possible functions; 3. Provide item writing and accessibility considerations; and 4. Provide the California Department of Education (CDE) with a reference guide to use while reviewing items.

Contents Table 1: Description of Item Types Table 2: Examples of Item Types

Description of Item Types Description of Item Functionality and Other Item Writing and Accessibility Item Type Characteristics Considerations MCSS This item type generally consists of a stem and list of A text-only version of this item type is choices; the test taker can select only one choice to accessible. (Multiple Choice, Single respond. The options use a radio button, but the test Select) taker can select text or an image to respond to the item by clicking in the option space rather than exactly in the radio button. Item should have four options: one key and three distractors. MCMS This item type generally consists of a stem and a list of A text-only version of this item type is choices; the test taker can select one or more choices accessible. (Multiple Choice, (options) to respond. The options use check boxes or Multiple Select) the test taker can select text or an image to respond to the item by clicking in the option space rather than exactly in the check boxes. The stem should specify how many selections should be made. A two-selection item should have four or five options, and a three- selection item should have five options.

Some item types are more accessible than others … Description of Item Item Writing and Item Type Functionality and Other Accessibility Characteristics Considerations Interactive Unscored simulations This item type is used require the test taker to with performance tasks select inputs that and not with discrete provide certain outputs items. to answer one or more This item type is not questions. Scored accessible. simulations require the test taker to select inputs that are scored.

Example Items Item Type Test Taker Example Interaction and Keys The test taker selects one InlineChoiceListSS response from a drop- down menu to fill in the blank. The keys are numerical values given in the same order as the available selections. For example, a 1 represents the first choice in the list as the correct answer.

Enhancing Technology Enhanced Items • Add information about the relationship between specific content and the various item types. •Add information about the function/design of distractors for selected response items. •Create guidelines for rubric development for short constructed response items. •Provide information on content that should not be included in specific item types.