Experimental Design & Evaluation 14. Quiz 3 SunyoungKim,PhD - PowerPoint PPT Presentation

Experimental Design & Evaluation 14. Quiz 3 Sunyoung�Kim,�PhD

1. Scientists always try to maximize reliability and validity in their research. a. True b. False 2. Reliability refers to whether we are truly measuring the concept of interest in our study. a. True b. False 3. The validity of a measure refers to the: a. accuracy with which it measures the variable. b. particular type of construct specification. c. consistency of the measurement. d. comprehensiveness with which it measures the construct.

4. Which of the following statements most likely involves using data in a way that is valid? a. University data indicate that the highest paid professors have the most publications. b. Students who can type more than 80 words per minute will receive a typing award. c. All of the teachers interviewed described the principal as "outstanding.” d. The English teacher does not assign students a grade less than "B.” 5. The extent to which we can generalize the results of a study to other participants is called: a. external validity b. internal validity

6. When you are confident that the experimental manipulation produced the changes you measured in the dependent variable, your study probably has good ________ validity. a. internal b. external 7. A measure has high internal consistency reliability when: a. participants score at the high end of the scale every time they complete the measure. b. multiple observers make the same ratings using the measure. c. multiple observers obtain the same score every time they use the measure. d. each of the items correlates with other items on the measure. 8. Double blind experiments increase experimenter bias. a. True b. False

9. What is the name of an experimental bias that refers to the tendency for performance to initially improve when new technology is instituted, not because of any actual improvement in learning or achievement, but in response to increased interest in the new technology? Novelty effect

10. What type of experimental bias does the experiment below suffer from? One study was performed at a factory to discover if a change in lighting would affect productivity. Productivity did increase but only because of the increased attention due to the study. As soon as the study had ended, productivity decreased again. In this example, the change in productivity had nothing to do with the lighting but rather the existence of the study in the first place. Hawthorne effect

11. Describe advantages and risks of A/B testing Advantages Subjects decisions to data rather than HiPPO (Highest Paid Person’s Opinion) • Great for optimizing copy, colors, placements, shapes, imagery • Can even (in theory) be used to optimize some elements automatically (text • colors, link styles You can be running hundreds simultaneously • Risks Hard to test completely new products, features • If many many aspects are subject to fluctuation, what is the ‘actual’ site? • Make sure you are asking the right questions! •

12. Define Cognitive Walkthrough with three questions to ask A usability evaluation method in which one or more evaluators work through a series of tasks and ask a set of questions from the perspective of the user. The focus of the cognitive walkthrough is on understanding the system's learnability for new or infrequent users 1. Visibility: Is the control for the action visible to the user? 2. Affordance: Is there a strong link between the control and the action? (Will the user notice that the correct action is available?) 3. Feedback: Is feedback appropriate? (Will the user properly interpret the system response?)

13. Describe advantages and risks of Heuristic Evaluation Advantages It’s fast, quick and cheap to conduct a heuristic evaluation. This is • especially true if you are only going to use a single evaluator. It provides good insight into possible usability problems that might • damage the user experience. Problems A single evaluator may miss issues that are not readily apparent to them. • The use of more than one evaluator is recommended. Heuristic analysis is subjective. It does not “prove” anything and thus • findings may be open to debate. Experienced evaluators are hard to come by and that means that you may • need to use less skilled evaluators whose findings may not be as valuable.

14. Describe a structure of report by explaining what question(s) each section needs to answer a. Introduction: “What is the problem?” and “Why should anyone care?” b. Methods: “What did I do?” c. Results/findings: “What did I find?” d. Discussion: “What does all of this mean?” and “Why does it matter?” e. Conclusion: a restatement of your work