Evaluating Potential Benefits and Risks of Technology Enhanced (TE) - PowerPoint PPT Presentation

Evaluating Potential Benefits and Risks of Technology Enhanced (TE) Items in Early Literacy and Mathematics Assessments: Results and Lessons Learned Julie Murphy Nichole Mosser K-5 Education Assessment Specialist English Language Arts Assessment Consultant Michigan Department of Education Michigan Department of Education Joseph Martineau Kyle Ward Senior Associate Mathematics Assessment Consultant Michigan Department of Education Center for Assessment June 26, 2019 CCSSO National Conference on Student Assessment This work is licensed under a Orlando, FL Creative Commons Attribution 4.0 International License.

Rationale “Hey Napoleon, what did you again last summer?” —Napoleon Dynamite www.nci 2 ea.org

Considerations in Moving Toward TEIs • The primary reason to use TEIs is to improve the directness with which items measure the intended standard(s) § This is particularly so for alignment to cognitive complexity, practices, and processes. § Several verbs used in content standards are problematic for traditional items, for example • Draw • Graph • Explain • Write • Demonstrate fluency... § When an examinee has to apply their knowledge by building a response, it is arguably a more direct measure than selecting a response from among a few of the possible response 3

Considerations in Moving Toward TEIs • TEIs can reduce indirectness of measuring more complex skills, but... § They do not necessarily reduce indirectness of measuring complex skills. § It is generally more difficult to write items that directly measure higher level skills, no matter the item type. • In my experience, many TEIs written to measure higher complexity content are de facto multiple choice items. § This may be problematic if the affordances of TEIs could have been used to more directly measure the intended content. § This may not be problematic if the intended content can be appropriately measured by a multiple choice question, or at least as directly as is possible with a TEI 4

Considerations in Moving Toward TEIs • There are two additional possibilities when comparing TEI item types with traditional item types § A TEI can improve on traditional item types in other ways than improving the directness of measuring complex content, for example... • Increasing examinee engagement • Consistency with increased classroom use of electronic devices • Use of color in graphics • Faster results § A TEI can introduce construct irrelevant variance • Requiring a drag and drop when a simple click would do • Unnecessary use of graphics or animation • Lack of attention to colorblindness • Anxiety on the part of stakeholder because of disrupting the status quo 5

Considerations in Moving Toward TEIs • It is important to evaluate whether the generally increased cost of TEIs be evaluated against the benefits and risks of using TEIs • When an assessment is already in place, this can be an anxiety- inducing task § There are many questions we don’t ask because we are not sure we want to know the answers § Do I want to identify a potential problem with an existing assessment. The implications are considerable § Think of how we do bias and sensitivity review vs. content review • We do data free review for both content review and bias/sensitivity review • We do data review for both content review and bias/sensitivity review • We generally do post-administration analysis only for content issues 6

What the Items Look Like “[They] will ride eternal, shiny and chrome” —Mad Max, Fury Road www.nci 7 ea.org

Explanation of Technology Enhanced (TE) Items • According to the Smarter Balanced Assessment Consortium, a Technology-Enhanced Item is a computer-delivered item which includes specialized interactions for collecting response data. These include interactions and responses beyond traditional selected- response or constructed-response.

Choice

Choice

Hot Spot

Hot Spot

Hot Text

Drag and Drop

Multiple Drag and Drop

Multiple Drag and Drop and Hot Spot

Methodology “Orthogonalization algorithm. Yes, sir. I prefer it over Euclidean coordinates.” —Hidden Figures www.nci 17 ea.org

Analysis of Questions Asked About TE Items About 800 ELA TE items and 640 Math TE items were reviewed considering four questions. The questions were: • Is the TE item a dressed-up Multiple Choice (MC) item? • If so, is the TE item OK as a dressed-up MC item (based on alignment to content standards and Depth of Knowledge (DOK)? • Excluding issues of alignment, does being a TE item introduce other helpful features beyond what would be possible using paper & pencil MC items? • Excluding issues of alignment, does being a TE item introduce features likely to create construct irrelevant variance that would not exist with paper & pencil MC items?

Analysis of Questions Asked About TE Items The possible options for each category were: • Definitely Not • Probably Not • Unsure • Probably Yes • Definitely Yes

Examples: Multiple Drag and Drop Not a dressed up MC item (requires multiple matching rather than selecting correct responses)

Examples: Multiple Drag and Drop Not a dressed up MC item (requires constructing a response rather than selecting correct responses)

Examples: Multiple Connection Matching Match each animal to the words that tell how it moves. Not a dressed up MC item (requires multiple matching rather than selecting correct responses)

Examples: Multiple Matching Table Not a dressed up MC item (requires multiple matching rather than selecting correct responses)

Order Not a dressed up MC item (requires reordering text rather than selecting correct responses)

Multiple Sequence Drag and Drop A student is writing a story about Mike Mouse. The student wants to add more details. Move 2 sentences into the paragraph that add the best details about Mike Mouse. A dressed up MC item (to select two responses, the student has to drag them to a certain spot)

Hot Text The sentence below has a mistake. Choose the word in the sentence that should begin with a capital letter. Not a dressed-up MC item (the answer is given in context rather than naked as a response option)

Line Plot Input Not a dressed up MC item (requires examinee to build a frequency line plot)

Hot Spot-Count Not a dressed up MC item (requires one-to-one correspondence as evidence by mouse clicks)

TEIs Introducing Features Likely to Create Construct Irrelevant Variance • There were some different reasons why TE items introduce features to create construct irrelevant variance that would not exist with paper and pencil multiple-choice items. Here are a few reasons. • Connection matching is not intuitive to the student. • Some students may not have the computer skills to interact with this type of TE item. • Some students may not know how to use technology to match the items.

TEIs Introducing Other Helpful Features • This assessment does not have a Paper/Pencil option. Therefore, our answer choices are based on students not transferring answer choices. • Consider- Paper/Pencil assessments include answer documents that students must use to transfer selections. For younger students, fine motor/vision transfer may be difficult. • Color and graphics play an important role in kindergarten students' ability to understand texts. Paper/Pencil assessments do not offer color as an option.

TEIs Introducing Other Helpful Features • ELA § There were 45 different reasons why the TE items introduce other helpful features beyond what would be possible using paper and pencil MC items. § Here are a few reasons. § Grade 2 • Drag and Drop Items § In context—paragraph, sentence. For example, it is helpful to move and then see a word, letter, picture, phrase, punctuation, etc. within the context. § Ordering—sequencing events. For example, it is helpful to move and then see the sentences in the correct order.

TEIs Introducing Other Helpful Features • Grade 2 (continued) § Matching Items—picture to text, text to text. For example, it would be helpful because the student can see the line going to the answer and eliminate options. § Hot Text Items—too many options. For example, there are too many words, phrases, sentence to create a multiple- choice question.

TEIs Introducing Other Helpful Features • Grade 1 has many of the same helpful features as in grade 2. • Here are some unique features. § Drag and Drop Items • In context—paragraph, sentence. For example, it is helpful to move and then see a punctuation at the end of a sentence or a capital letter at the beginning. • Graphic with text—paragraph, sentence, word. For example, it is helpful to move and then see the description next to the picture or sequence the pictures. § Hot Text • When the word is read to the student, the item is more user friendly and the pronunciation is consistent. • It is helpful to be able to click on the first word in the sentence as opposed to multiple choice in paper and pencil.