Experimental Design & Evaluation 10. Controlled Experiment - PowerPoint PPT Presentation

Experimental Design & Evaluation 10. Controlled Experiment Sunyoung�Kim,�PhD�

Last week Prototyping •

Recap. What is Prototyping? Prototypes are experimental and incomplete designs which are • cheaply and fast developed An integral part of iterative user-centered design • It enables designers to try out their ideas with users and to gather • feedback

Recap. Sketches vs. Prototypes Sketches • Early ideation - stages of design Exploring ideas - Prototypes • Capturing/detailing - the actual design Testing ideas -

Recap. Wireframe (Mid-fi prototype) Using computer-based tools (e.g., balsamiq, wireframe.cc) • Take more time and effort but look more formal and refined: more • detailed than sketches Interactivity can be simulated • à You don’t need to make these things pretty but you do need to include enough detail to see how the system performs à Force users to view it as a draft or work in progress, rather than a polished and finished product à Prototype a high visual fidelity (e.g., done in Photoshop) makes the user to focus on the visual design and look and feel, including color, fonts, layout, logo and images

Recap. High fidelity The most realistic but time-intensive • The only way to create high-fidelity prototypes used to actually code • using a programming language; these days, you can create high-fidelity prototypes that simulate the Functionality of the final product without coding (e.g., Axure, iRise, omni graffle) Appropriate when high visual and functional fidelity is required • An excellent reference for developers • Tools: https://www.cooper.com/prototyping-tools •

Recap. Wizard of Oz • A rapid-prototyping method for systems costly to build or requiring new technology. A human “Wizard” simulates the system’s intelligence and interacts with the user through a real or mock computer interface. • Makes it possible to test functionality that does not yet exist • Can simulate different system behaviors and test result (e.g., speed of ticket from input to output) • Can simulate errors and test result • Common in areas such as intelligent agents, human- robotic interaction

Today’s agenda Controlled experiments • Hypothesis testing • Threats •

Controlled Experiment

HCI Research Methods • Controlled (lab) experiment • Field Experiment

HCI Research Methods • Controlled (lab) experiment • Create a situation with desired conditions • Manipulate some variables while controlling others • Examine the dependent variable • Field Experiment • Conduct study in a natural setting • Manipulate some variables • Controlling other variables won’t be permitted • Examine the dependent variables

Controlled Experiment A test of the effect of a single variable by changing it while keeping all other variables the same. A controlled experiment generally compares the results obtained from an experimental sample against a control sample.

Controlled Experiment 1. Create a situation with desired conditions 2. Manipulate some variables while controlling others 3. Examine the dependent variable

Designing an Experiment 1. State a research question(s) 2. State a testable hypothesis 3. Identify independent and dependent variables 4. Design the experimental protocol 5. Choose the user population 6. Run an experiment 1) Manipulate an independent variable 2) Measure dependent variables 3) Use statistical tests to accept or reject the hypothesis

Protocol

Research Question It's essential to develop a research question to focus your research. 1. Choose an appropriate topic or issue for your research 2. List all of the questions that you'd like answered yourself 3. Choose the best question, one that is neither too broad nor too narrow What is the 1994 rate of juvenile delinquency in the U.S.? • What can we do to reduce juvenile delinquency in the U.S.? • Does education play a role in reducing juvenile delinquents' return • to crime?

Example Research Questions How does the temperature of sea water affect the amount of • calcium carbonate that can be dissolved in it? What can be done to stop the pH of the ocean changing? • How does the amount of light influence the rate of algal • growth? Do paper bags biodegrade faster than plastic grocery bags? • What type of packaging preserves antioxidant activity in food • the best? How can chemicals be used to reduce the spread of bacteria? •

Hypothesis A statement of the predicted or expected relationship between at least two variables A provisional answer to a research question • Has to define the variables involved • Has to define a relationship • Example • - Research question: Will a lower pH of seawater increase the rate of ice melting? - Hypothesis: An increase in the number of ions in solution will increase the rate that water molecules move from a solid into a liquid state

Hypothesis A statement of the predicted or expected relationship between at least two variables Relationship Variable 1 Variable 1 • Research question: How does having information on the context of a caller affect whether the receiver picks up the call? • Hypothesis: Receivers will be more likely to pick up a call when they have information of their callers’ context than they will be when they do not. Improve Information of Call pickup caller’s context

Good Hypothesis Testable: The means for manipulating the variables and/or • measuring the outcome variable must potentially exist Falsifiable: Must be able to disprove the hypothesis with data • Parsimonious: Should be stated in simplest adequate form Precise: • Should be specific (operationalized) Useful: Relate to existing theories and/or “point” toward new • theories. It should lead to studies beyond the present one (often hard to determine in advance)

Hypothesis “iPad is better than Kindles”: Is it testable hypothesis?

Hypothesis “iPad is better than Kindles”: Is it testable hypothesis? No! because: • Broad questions are not testable • Broad questions can be investigated by posing multiple narrow testable questions

Hypothesis “iPad is better than Kindles”: Is it testable hypothesis? No! because: You are unclear with • What feature? • What task? • What measurement? • What population?

Hypothesis “College students (population) type (task) faster (measurement) using iPad’s keyboard (feature) than using Kindle’s keyboard” * Can still be even more narrow e.g., in a classroom

Variables Relationship Variable 1 Variable 2 Independent variable: Dependent variable: What you manipulate What you measure

Variables Independent variable: Things we want to compare • Dependent variable: Things we want to measure • Confounding variable: Things that correlate with the independent • + dependent variable

Independent Variable IV. What you very • Independent of participant behavior • Examples: interface, visual layout, gender, age • Test conditions: levels, or value of an IV • Provide a name for both IV and its levels (test conditions) •

Dependent Variable DV, what you measure • o User performance time o Accuracy, errors o Subjective satisfaction

Confounding Variable A confounding variable is one that provides an alternative • explanation for the thing we are trying to explain with our IVs. Example: we want to compare two systems (windows 7 vs. 8) • o All participants have prior experience with windows 7, but no experience with windows 8 o “Prior experience” is a confounding variable A major issue in observation studies is that we often don't always • know what the potential confounding factors may be.

Example Research Questions How does the temperature of sea water affect the amount of • calcium carbonate that can be dissolved in it? What can be done to stop the pH of the ocean changing? • How does the amount of light influence the rate of algal • growth? Do paper bags biodegrade faster than plastic grocery bags? • What type of packaging preserves antioxidant activity in food • the best? How can chemicals be used to reduce the spread of bacteria? •

Hypothesis “College students (population) type (task) faster (measurement) using iPad’s keyboard (feature) than using Kindle’s keyboard” Independent variable • Dependent variable • Control variable • Confounding variable •

Hypothesis “College students (population) type (task) faster (measurement) using iPad’s keyboard (feature) than using Kindle’s keyboard” Independent variable: device (iPad or Kindle) • Dependent variable: typing speed • Control variable: College students • Confounding variable: • Prior technology experience

Hypothesis testing Causality vs. Correlation Causal: One variable depends on and is affected by the other • Correlational: Two variables are affected by a third variable in the • same direction

Causality vs. Correlation Dependent variable Variable 2 Correlation Variable 1 Independent variable Variable 3 Dependent variable

Causal

Correlational

Causality vs. Correlation Dependent variable Variable 2 Correlation Variable 1 Independent variable Variable 3 Dependent variable