Foundations of experimental research 707.031: Evaluation Methodology Winter 2014/15 Eduardo Veas
THEOC, the scientific method Theory Hypothesis Experiment Observation Conclusion 2
Source of variability Source: Card et al 1983 3
Curiosity Human behaviour foundation of experimental research 4
Ideas, Theories and Hypothesis Curiosity in motion 5
Experimental research • Establish relationships between circumstances and behaviors • Fit these relationships into an orderly body of knowledge 6
FEAR OF IDEAS 7
Fear your ideas Anyone doing research is a genius, I don’t come even close GENIEPHOBIA 8
Fear your ideas I am having a hard time coming with original ideas IMITATOPHOBIA 9
Fear your ideas Having to use complex hardware… I got a headache PARAPHERNALIO-PHOBIA If there is complex equipment involved, it must be good research MANUPHOBIA 10
Fear your ideas If it is simple it can’t be science FEAR OF SIMPLICITY 11
Fear your ideas numbers! numbers! FEAR OF STATISTICS 12
Fear your ideas Something is missing here, I just know it IMPERFECTAPHOBIA 13
Fear your ideas Lorem ipsum… FEAR OF NOT SOUNDING SCIENTIFIC 14
Fear your ideas FEAR OF WORK 15
Fear your own ideas • Geniephobia • Imitatophobia • Paraphernalio-phobia / Manuphobia • Fear of simplicity • Fear of math • Imperfectaphobia • Fear of not sounding scientific • Fear of work 16
Generating ideas Systematic reduction of idea-phobia 17
Experimental research • Establish relationships between circumstances and behaviors • Fit these relationships into an orderly body of knowledge 18
Observation • Sit in your computer and stare at your keyboard until your eyes start to bleed 19
Observation • Sit in your computer and stare at your keyboard until your eyes start to bleed • We are interested in human, rather than keyboard behavior 20
Public observation write up ideas that come up as you stroll through campus. 21
Public observation write up ideas that come up as you stroll through campus. you got 7 minutes. statements in the form circumstance => behavior 22
ROT test Experimental ideas must be: • R epeatable • O bservable • T estable 23
Correlational or observational? • Label your ideas now 24
Theories Title Text 25
Theory induction • choose one idea and convert it into a theory deduction • use that theory to make predictions • each prediction forms a hypothesis 26
Relationship Theory-Hypothesis-Experiment Observation induction Theory deduction Predicted Observation EXPERIMENT Confirmed Observation Disconfirmed Observation induction deduction Theory supported Theory false 27
Expected results of experiment • proving a prediction: does not prove but supports a hypothesis, thus the theory. • disproving a prediction: not enough evidence was found to prove the hypotheses/theory 28
Does theory precede data? OBSERVATION EXPERIMENT THEORY HYPOTHESES 29
Experimental Methodology Formal Curiosity 30
Experimental research • Establish relationships between circumstances and behaviors • Fit these relationships into an orderly body of knowledge 31
Experimental research • Behavior Circumstances 32
Experimental research grab more heavy breakfast coffee hot office press button bad coffee bright office read email rainy day increase light intensity birthday party text girlfriend last night loud pitch smelly sound office text boyfriend sunny day Circumstances Behavior 33
Experimental research grab more heavy breakfast coffee hot office press button bad coffee bright office read email rainy day increase light intensity birthday party text girlfriend last night loud pitch smelly sound office text boyfriend sunny day Circumstances Behavior 34
Experimental research: causal statements Increase in light press button intensity WHEN DONE CORRECTLY change in measured behavior is due to manipulation of circumstance 35
Variables in experimental research Title Text 36
Experimental research: variables INDEPENDENT press button increase light intensity Circumstances Behavior 37
Experimental research: variables INDEPENDENT DEPENDENT press button increase light intensity Circumstances Behavior 38
Experimental research: hypothesis INDEPENDENT DEPENDENT press button increase light intensity the hypothesis is a statement about the expected outcome Circumstances Behavior 39
Experimental research: hypothesis INDEPENDENT DEPENDENT press button increase light intensity H1: Participants will be significantly faster in pressing a button in the 100 Lux condition. Circumstances Behavior 40
Experimental research: variables INDEPENDENT DEPENDENT heavy breakfast hot office press button bad coffee bright office rainy day increase light intensity birthday party last night loud pitch smelly sound office -?- sunny day Circumstances Behavior 41
Experimental research: variables INDEPENDENT DEPENDENT heavy breakfast hot office press button bad coffee bright office rainy day increase light intensity birthday party last night loud pitch smelly sound office -control variables- sunny day Circumstances Behavior 42
Experimental research: external validity • validity of experimental method: is drawing conclusions about cause justifiable? • the more highly controlled the experiment, the less generalizable its results. 43
Experimental research: variables INDEPENDENT DEPENDENT heavy breakfast hot office press button bad coffee bright office rainy day increase light intensity birthday party last night learning loud pitch smelly sound office -control variables- sunny day Circumstances Behavior 44
Experimental research: variables INDEPENDENT DEPENDENT heavy breakfast hot office press button bad coffee bright office rainy day increase light -random variables- intensity birthday party last night learning loud pitch smelly sound office -control variables- sunny day Circumstances Behavior 45
Experimental research: random variables • random selection of participants. • random assignment of circumstances to levels of the independent variable 46
Experimental research: variables generalizable RANDOM CONTROL RANDOM WITHIN CONSTRAINTS 47
Experimental research: variables circumstance that changes systematically as the experimenter manipulates the independent variable CONFOUNDING VARIABLES 48
VALIDITY Where we see it all fail 49
Experimental Method: validity • External: is it justifiable to generalize causation from the results • Internal: are there confounding variables which have not been taken into account? are there unconsidered threats? 50
Threats to internal validity • History • Maturation • Selection • Differential mortality • Testing • Statistical regression • Interactions with selection 51
Experimental Design Blueprint of a proof 52
Experimental design questions • HOW MANY INDEPENDENT VARIABLES? • HOW MANY DIFFERENT VALUES DOES EACH VARIABLE HAVE? 53
Experimental design: Decision Design study Number of independent variables >1? NO YES Basic Factorial design design Number of values per independent variable Between Within Between Within Split plot groups groups groups groups 54
Between participant design • Each participant is exposed to one level only • Divide participants in groups (one per condition) • Compare measurements between groups 55
Between participants design: advantages • Rules out learning effects. • Cannot contaminate behavior in other levels • Can collect more data per level / more participant time per level. 56
Between participant design: disadvantages • differences between groups of participants = differences between conditions 57
Within participants design • Each participant is exposed to every condition 58
Within participant design: advantages • requires fewer participants • minimizes individual differences between levels of the independent variable 59
Within participant design: disadvantages • Needs to account for learning effects • Needs to account for ordering effects • Combinatorial explosion limits number of conditions 60
Number of participants • Depends on • effect size • study design • Calculated through power analysis (statistical power) 61
Experimental design: overview • Trial: independent unit of measurement • Measurement: • quantitative: measurable indicators (task completing time, error rates, mouse movement) • qualitative: subjective feedback (satisfaction, preference) • observations 62
Recommend
More recommend
