Test-item Writing Workshop: The Power of Peer-review October 4, - PowerPoint PPT Presentation

Test-item Writing Workshop: The Power of Peer-review October 4, 2017 Fidel Lim, DNP, CCRN

Objectives • Identify 4 flaws in test questions • Utilize best practices for writing multiple choice questions • Utilize best practices when writing alternate items • Correctly classify questions based on Bloom’s Taxonomy 2 1/12/2018

What are we testing? • An effective test item: • Discriminates students who understand the content from those who do not. • Focuses on important information. 3

Four Categories of Test-item Flaws 1. Test-wise flaws - Flaws that make it easier for “ test-wise ” students to answer the question correctly, giving them an unfair advantage over students who are not test- wise. (3%) (Option length, grammatical errors in options that are not correct answer, repetition of key word in correct answer, “ all of the above ” ) 4 Presentation Title Goes Here

Four Categories of Test-item Flaws 2. Irrelevant difficulty - Flaws in either the stem or options that make items difficult to understand for reasons unrelated to the content or focus of the assessment. Difficult to understand = completion format 5 Presentation Title Goes Here 1/12/2018

Four Categories of Test-item Flaws 3. Linguistic/structural bias - Unnecessary linguistic complexity in the stem or options, grammatical errors, and lack of clarity or consistency in the wording. 6 Presentation Title Goes Here 1/12/2018

Four Categories of Test-item Flaws 4. Cultural bias - Use of culturally specific information that is not equally available to all cultural groups. (Questions with information not equally available to all cultural groups – “ gin & tonic ” ) 7 Presentation Title Goes Here 1/12/2018

2014 RN Practice Analysis: Linking the NCLEX-RN Examination to Practice • A practice analysis study aims to investigate the applicability of the 2013 NCLEX-RN Test Plan. This periodic performance of practice analyses (i.e., job analysis) studies assists NCSBN in evaluating the validity of the test plan that guides content distribution of the licensure examination. https://www.ncsbn.org/15_RN_Practice_Analysis_Vol62_ web.pdf 8 Presentation Title Goes Here 1/12/2018

Applicability of Activities to Practice Setting Lowest percentage of applicability • “Provide care and education to a client in labor and antepartum client” (21.5%), • “Provide post-partum care and education” (23.0%) • “Provide care and education for the newborn less than 1 month old through the infant or toddler client through 2 years” (26.7%). 9 Presentation Title Goes Here 1/12/2018

Applicability of Activities to Practice Setting The activities with the highest percentage of applicability: • “Organize workload to manage time effectively” (100.0%) • “Apply principles of infection control (e.g., hand hygiene, surgical asepsis, isolation, sterile technique, universal/standard precautions)” (100.0%). 10 Presentation Title Goes Here 1/12/2018

Frequency of Activity Performance Setting-Specific The activities performed with the lowest frequencies: • “Implement emergency response plans (e.g., internal/external disaster, bomb threat, community planning)” (0.67), • “Implement and monitor phototherapy” (0.67), • “Perform hemodialysis or continuous renal replacement therapy” (0.68), and • “Perform and manage care of client receiving peritoneal dialysis” (0.70). 11 Presentation Title Goes Here 1/12/2018

Frequency of Activity Performance Setting-Specific The activities performed with the highest frequencies: • “Apply principles of infection control (e.g., hand hygiene, surgical asepsis, isolation, sterile technique, universal/ standard precautions)” (4.94), • “Provide care within the legal scope of practice” (4.82), and • “Ensure proper identification of client when providing care” (4.82). 12 Presentation Title Goes Here 1/12/2018

Frequency of Activity Performance Total Group The activities performed with the lowest frequency were • “Implement and monitor phototherapy” (0.20), • “Perform hemodialysis or continuous renal replacement therapy” (0.22), and • “Provide post-partum care and education (0.31). 13 Presentation Title Goes Here 1/12/2018

Frequency of Activity Performance Total Group The activities performed with the highest frequencies were: • “Apply principles of infection control (e.g., hand hygiene, surgical asepsis, isolation, sterile technique, universal/ standard precautions)” (4.94), • “Provide care within the legal scope of practice” (4.82), and • “Ensure proper identification of client when providing care” (4.80). 14 Presentation Title Goes Here 1/12/2018

Importance of Activity Performance Setting-Specific The activities with the lowest importance ratings were: • “Recognize complementary therapies and identify potential contraindications (e.g., aromatherapy, acupressure, supplements)” (3.52), • “Plan and/or participate in community health education” (3.55), and • “Perform post-mortem care” (3.73). 15 Presentation Title Goes Here 1/12/2018

Importance of Activity Performance Setting-Specific The activities with the highest importance ratings were:  “Apply principles of infection control (e.g., hand hygiene, surgical asepsis, isolation, sterile technique, universal/standard precautions)” (4.91),  “Prepare and administer medications, using rights of medication administration” (4.89), and  “Ensure proper identification of client when providing care” (4.88). 16 Presentation Title Goes Here 1/12/2018

Importance of Activity Performance Total Group The activities with the lowest importance ratings were: • “Recognize complementary therapies and identify potential contraindications (e.g., aromatherapy, acupressure, supplements)” (3.31), • “Plan and/or participate in community health education” (3.37), and • “Implement and monitor phototherapy” (3.42). 17 Presentation Title Goes Here 1/12/2018

Importance of Activity Performance Total Group The activities with the highest importance ratings were: • “Apply principles of infection control (e.g., hand hygiene, surgical asepsis, isolation, sterile technique, universal/standard precautions)” (4.91), • “Ensure proper identification of client when providing care” (4.88), and • “Prepare and administer medications, using rights of medication administration” (4.88). 18 Presentation Title Goes Here 1/12/2018

Bloom’s Taxonomy Remembering: define, duplicate, list, memorize, recall, Can the student recall or remember the repeat, reproduce state information? Understanding: classify, describe, discuss, explain, identify, Can the student explain ideas or locate, recognize, report, select, translate, concepts? paraphrase Applying : choose, demonstrate, dramatize, employ, Can the student use the information in a illustrate, interpret, operate, schedule, sketch, new way? solve, use, write. Analyzing : appraise, compare, contrast, criticize, Can the student distinguish between the differentiate, discriminate, distinguish, different parts? examine, experiment, question, test.

Remembering Level • Requires recall or recognition of information previously learned. • Question requires no understanding or judgment. • Good for testing factual information. • Limited value for testing nursing competence.

Understanding Level • Evaluates a student ’ s understanding of the information. Examples • Pathophysiology (e.g., you do this because …, why do you do something) Behavioral objective: • Explain the cause of complications of diagnostic procedures for the liver.

Applying Level • Requires students to apply their knowledge in a situation. • Tests ability to use learned concepts in specific situations. • This is where you are asking the student to make some sort of judgment. Examples: • Evaluating client responses or statements. • Determining which client situation requires immediate action.

Analyzing Level • Requires students to break down information to obtain the meaning of the material or demonstrate how it is organized or structured. • Requires students to analyze the situation, interpret data, set goals, establish nursing diagnoses, set priorities, and determine actions. • This is a multiple step process – the student has to figure out what ’ s going on with the patient and then what to do about it.

Question Stem Should Should not: Contain a subject and a verb Contain irrelevant information CAPITALIZE, italicize, or underline “ Teach" information qualifier word to support the critical thinking process Always be a client and a nurse

Write the Stem: Example Stem Joe, aged 25, is admitted to the hospital with a ruptured appendix. His temperature is 101 ° F and WBC is 15,000. During his post-op recovery phase Joe receives TPN and complains of thirst and frequent urination. This complaint most likely indicates:

Example Stem Good - The nurse cares for the client receiving TPN. The patient reports increased thirst and frequent urination. These symptoms MOST likely indicate which complication?

Write the Correct Answer/s • Write and validate the correct response/s • Vary the position of the correct response/s • Should be single action