Making Multiple Choice Tests More Effective Linda Suskie Assessment & Accreditation Consultant Penn State University April 25, 2017
Handouts 1. Intro to Statistics Test Blueprint 2. Educational Research Methods Final Exam Outline 3. Quiz 2 ▪ 1 st item: “At the moment, which of the following is the smallest?” 4. Tips on Writing Good Multiple Choice Questions 5. Exam Analysis Report in color 6. Quiz 2 ▪ 1 st item” “What kind of increases in the price level are most damaging to the economy?”
Today… ▪ Design your tests to address course objectives. ▪ Design your tests to be of appropriate difficulty. ▪ Minimize barriers and clues in your test questions. ▪ Use item analyses to identify areas for improvement in test questions and student learning.
Vocabulary ▪ Stem ▪ Direct question ▪ Incomplete sentence ▪ Alternatives/responses/options ▪ Answer ▪ Distracters/foils
Design your tests to address your course’s important objectives.
Use test blueprints. ✓ Fair and balanced test (content validity) ▪ Content areas ▪ Memorization vs thinking skills ✓ Study guide ✓ Useful results
ECON 104 Section 2 Quiz 2 Test Blueprint? Number of Learning Objective Items 9 Knowledge & Conceptual Understanding 3 Know current and recent economic indicators and characteristics (Items 1, 5, 9) 2 Define and give examples of economic terms (Items 2, 12) 2 Explain the impact of economic changes (Items 7, 8) 1 Explain the impact of changes in Federal economic policy (Item 3) 5 Thinking Skills (analyze, evaluate, apply, etc.) 5 Calculate economic indicators (Items 4, 6, 10, 11, 14)
Design your tests to be of appropriate difficulty. ▪ Which objectives are essential, foundational skills? ▪ All students passing the course should be able to do these. ▪ Virtually everyone should answer these correctly ▪ Which objectives are aspirational…challenging? ▪ These separate the A students from the C students.
Barriers To students who should answer correctly
Don’t penalize slow readers. Keep items concise. Minimize unnecessary reading. If firms in a monopolitistically competitive market are currently earning economic profits, then in the long run, A. New firms will enter the market, and the current firms will experience a decrease in demand for their products until zero economic profit is again restored. B. New firms will enter the market, and the current firms will experience an increase in demand for their products until zero economic profit is again restored. C. Some existing firms will exit the market, and the remaining firms will experience an increase in demand for their products until zero economic profit is again restored. D. Some existing firms will exit the market, and the remaining firms will experience a decrease in demand for their products until zero economic profit is again restored. Firms in a monopolistically competitive market are currently earning economic profits. Which statement best describes what will happen in the long run, as the market changes, until zero economic profit is again restored? A. New firms will enter the market and current firms will see decreased demand. B. New firms will enter the market, and current firms will see increased demand. C. Some existing firms will exit the market, and remaining firms will see increased demand. D. Some existing firms will exit the market, and remaining firms will see decreased demand.
The stem should ask a complete question. In perfectly competitive markets, firms A.can sell all of their output at the market price. B. produce differentiated products. C. can influence the market price by altering their output level. D.are large relative to the total market. E. try to maximize revenue.
Students should not have to read all options to understand and answer the question. The central meridian of UTM zone 31 is 3 E. The position of Greenwich, England, in geographic coordinates is 0 E, 51 N. What is the position of Greenwich in UTM coordinates, relative to zone 31? (You should be able to identify the correct answer by eliminating those that are obviously incorrect.) A.291,707 m E, 5,707,844 m N B. -291,707 m E, 5,707,844 m N C. 0 m E, 5,707,844 m N D.0 m E, 0 m N
Be careful with “best answer” questions. ▪ Change “Which of the following” to “What.” Which of the following best defines opportunity cost? A.It is the cost of producing those goods most desired by a given economy. B. It is the cost of the input mix that will lead to the greatest rate of growth for a given company. C. It is the amount of one product that must be given up in order to produce an additional unit of another product. D.It is the use of the least-cost method of production. E. It is the cost of labor used in the production process.
Avoid “K - Questions” ▪ Really true-false statements strung together ▪ Confusing ▪ Tests logical reasoning ability more than what’s learned Which of the following changes in the demand curve for a monopolistically competitive firm’s product will be beneficial for that firm? A. Shift it to the right B. Make it more elastic C. Make it more inelastic D. A and B only E. A and C only
Clues That help students who should answer in correctly
Keep Options Roughly the Same Length ▪ If we observe firms earning zero economic profit in the short run, we know that A. The industry must be perfectly competitive. B. The industry must be either perfectly competitive or monopolistically competitive. C. There must not be any barriers to entry. D. Any market structure is possible since firms under any market structure can earn zero profits at some time. E. The firm is a monopoly.
Minimize chances of guessing correctly. 50% 40% 30% 20% 10% 0% 2 options 3 options 4 options 5 options (true/false)
Avoid true-false! (questions with only 2 options) ▪ 50% chance of guessing correctly ▪ Students can recognize a false statement without knowing the true/correct alternative. ▪ Mostly tests memorized knowledge ▪ Hard to write Which of the following will be most affected if gasoline prices suddenly shoot up? A. The core rate of inflation B. The headline rate of inflation
Avoid interlocking items. ▪ Students can deduce the correct answer by looking at how the options are used in other items. Choose the term that exemplifies the aspect of data quality described in the following statement: “Ninety percent of well- defined points tested are located within 0.02 inches (map distance) of their actual locations.” A. Accuracy B. Precision C. Resolution D. Validity Choose the term that exemplifies the aspect of data quality described in the following statement: “UTM coordinates specified to two decimal places (e.g., 500,000.00 E, 500,000.00 N denote a smaller area on the ground that UTM coordinates taken to only one decimal place (e.g., 500,000.0 E, 500,000.0 N).” A. Accuracy B. Precision C. Resolution D. Validity
Convert interlocking items into matching. Choose the term that exemplifies the aspect of data quality described in each statement. Options may be used more than once or not at all. 1. Ninety percent of well-defined A. Accuracy points tested are located within 0.02 B. Precision inches (map distance) of their actual C. Resolution locations. D. Validity 2. UTM coordinates specified to two decimal places (e.g., 500,000.00 E, 500,000.00 N denote a smaller area on the ground that UTM coordinates taken to only one decimal place (e.g., 500,000.0 E, 500,000.0 N).
Reviewing & Using Test Results Five Things to Think About
#1. Test blueprints and item difficulty identify learning objectives with poor student achievement. ▪ How many students got each question right? ▪ How many students did you want to get each question right? ▪ Celebrate the ones students did well on! ▪ Look hard at any questions that students did poorly on. ▪ “Suskie 50% rule” ▪ Set your own benchmark: 80% 60%?
Easiest on ECON 104 Section 4 Quiz 2 #1 97% #13 92% #12 92% #14 92%
Hardest on ECON 104 Section 4 Quiz 2 #8 49% #6 57% #9 61%
#2. Why did students do poorly? ▪ Design and wording of test questions ▪ Curriculum design ▪ Did students spend enough time on learning activities on this objective? ▪ Teaching methods ▪ Were the learning activities and resources well designed? ▪ Ask your students!
#3. Good distracters diagnose where students went wrong with their thinking. ▪ What is 2 .10? A. 20 B. 2.10 C. 0.2 D. 0.02
#4. Distracters that few students choose may be a waste of reading time. ▪ 13. In the latest report, the unemployment rate was __ and about __ jobs were added. A. 4.8%, 227,000 (191) B. 4.8%, 337,000 (55) C. 5.8%, 227,000 (7) D. 5.8%, 337,000 (7) ▪ But all attractive options are impossible in very easy items.
#5. Item discrimination identifies bad test items. ▪ Students who do best overall should be more likely to get any one item right . ▪ Students who do worst overall should be more likely to get any one item wrong . ▪ Questions that top students did worse on than bottom students are bad. ▪ Throw out and rescore. ▪ Easy items cannot discriminate well.
Recommend
More recommend
