Quantitative analysis with statistics (and ponies) (Some slides and - PowerPoint PPT Presentation

Quantitative analysis with statistics (and ponies) (Some slides and pony examples from Blase Ur) 1

Logistics and updates • New homework coming soon • Ethics reading for Thursday – Ethical or not? Come prepared to vote • No office hours today – By appointment this week instead 2

Statistics • The main idea: Hypothesis testing • Choosing the right test: Comparisons • Regressions • Other stuff – Non-independence, directional tests, effect size • Tools 3

What’s the big idea, anyway? OVERVIEW 4

Statistics • In general: analyzing and interpreting data • We often mean: Statistical hypothesis testing – Is it unlikely the data would look like this unless there is actually a difference in real life? 5

The prototypical case • Q: Q: Do ponies who drink more caffeine make better passwords? • Experiment: Recruit 30 ponies. Give 15 caffeine pills and 15 placebos. They all create passwords. http://www.fanpop.com/clubs/my-little-pony-friendship-is-magic/images/33207334/title/little-pony-friendship-magic-photo 6

Hypotheses • Nul Null hypot l hypothesis hesis: There is no difference Caffeine does not affect pony password strength. • Al Alternat ternative hypot ive hypothesis hesis: There is a difference Caffeine affects pony password strength. • Note what is not here (more on this later): – Which direction is the effect? – How strong is the effect? 7

Hypotheses, continued • Statistical test gives you one of two answers: 1. Reject the null: We have strong evidence the alternative is true. 2. Don’t reject the null: We don’t have strong evidence the alternative is true. • Again, note what isn’t here: – We have strong evidence the null is true. (NOPE) 8

P values • What is the probability that the data would look like this if there’s no actual difference ? – i.e., Probability we tell everyone about ponies and caffeine but it isn’t really true • Most often, α = 0.05; some people choose 0.01 – If p < 0.05 , reject null hypothesis; there is a “significant” difference between caffeine and placebo – True or false ONLY: You don’t say that something is “more significant” because the p-value is lower – A p-value is not magic, it’s just probability 9

P values and correction • Type I error (false positive) – You expect this to happen 5% of the time if α = 0.05 • What happens if you conduct a lot of statistical tests in one experiment? • Many methods for “correcting” p values – Bonferroni correction (multiply p values by the number of tests) is the easiest to calculate but most conservative 10

Type II Error (False negative) • There is a difference, but you didn’t find evidence – No one will know the power of caffeinated ponies • Hypothesis tests DO NOT BOUND this error • Instead, statistical power is the probability of rejecting the null hypothesis if you should – Requires that you estimate the effect size (hard) 11

Hypotheses, power, probability • After an experiment, one of four things has happened (total P=1). PROBABILITY You rejected the null You didn’t Reality: Difference Estimated via power analysis ? Reality: No difference Bounded by α ? • Which box are you in? You don’t know. 12

Correlation and causation • Correlation: We observe that two things are related Do rural or urban ponies make stronger passwords? • Causation: We randomly assigned participants to groups and gave them different treatments – If designed properly Do password meters help ponies? 13

CHOOSING THE RIGHT TEST 14

What kind of data do you have? • For explanatory and outcome variables • Quantitative – Discrete (Number of caffeine pills taken by each pony) http://i196.photobucket.com/albums/aa92/ – Continuous (Weight of each pony) karina408_album/Wallpaper-53.jpg • Categorical – Binary (Is it or isn’t it a pony?) – Nominal: No order (Color of the pony) – Ordinal: Ordered (Is the pony super cool, cool, a little cool, or uncool) 15

What kind of data do you have? • Does your dependent data follow a normal distribution? (You can calculate this!) http://www.wikipedia.org – If so, use parametric tests. – If not, use non-parametric tests. • Are your data independent? – If not, repeated-measures, mixed models, etc. 16

If both are categorical …. • Use (Pearson’s) χ 2 (Chi-squared) test of independence. – Fewer than 5 data points in any single cell, use Fisher’s Exact Test (also works with lots of data) • Do not use χ 2 if you are testing quantitative outcomes! 17

Contingency tables • Rows one variable, columns the other • Example: • χ 2 = 97.013, df = 14, p = 1.767e-14 18

Explanatory: categorical Outcome: continuous/ordinal …. • If you want to compare “Which is bigger?” • Normal, continuous outcome (compare mean): – 2 conditions: T-test – 3+ conditions: ANOVA • Non-normal data / ordinal data – Does one group tend to have larger values? – 2 conditions: Mann-Whitney U (AKA Wilcoxon rank-sum) – 3+ conditions: Kruskal-Wallis 19

Continuous/ordinal data 20

What about Likert-scale data? • Respond to the statement: Ponies are magical. – 7: Strongly agree – 6: Agree – 5: Mildly agree – 4: Neutral – 3: Mildly disagree – 2: Disagree – 1: Strongly disagree 21

What about Likert-scale data? • Some people treat it as continuous (not good) • Other people treat it as ordinal (better!) – Difference 1-2 ≠ 2-3 – Use Mann-Whitney U / Kruskal-Wallis • Another OK option: binning (simpler) – Transform into binary “agree” and “not agree” – Use χ 2 or FET 22

Password meter annoying Control baseline meter three-segment green tiny Visual huge no suggestions text-only bunny half-score one-third-score Scoring nudge-16 nudge-comp8 Visual & text-only half- score bold text-only half- Scoring score 23 23

Contrasts • If you have more than two conditions, H 1 = “the conditions are not all the same” – “Omnibus test” • If you reject this null, you may compare conditions • Planned vs. unplanned contrasts contrasts • N-1 free planned planned contrasts – Actually, really planned. No peeking at the data. • Unplanned and post-hoc require p-correction 24

Contrasts in the meters paper “We ran pairwise contrasts comparing each condition to our two control conditions, no meter and baseline meter. In addition, to investigate hypotheses about the ways in which conditions varied, we ran planned contrasts comparing tiny to huge, nudge-16 to nudge-comp8, half-score to one- third-score, text-only to text-only half-score, half- score to text-only half-score, and text-only half- score to bold text-only half-score.” 25

Continuous/ordinal data 26

Finding a relationship among variables REGRESSIONS 27

Regressions • What is the relationship among variables? – Generally one outcome (dependent variable) – Often multiple factors (independent variables) • The type of regression you perform depends on the outcome – Binary outcome: logistic regression – Ordinal outcome: ordinal / ordered regression – Continuous outcome: linear regression 28

Example regression • Outcome: – Completed pony race (or not): Logistic – Finish time in pony race: Linear • Independent variables: – Age of pony – Number of prior races – Diet: hay or pop-tarts (code as eatsHay=true/false) – (Indicator variables for color categories) – Etc. 29

What you get • Linear: Outcome = ax 1 + bx 2 + c – Finish time = 3*age - 5*eatsHay + 7 • Logistic: Outcome is in log likelihood – Intuition: probability of finishing decreases with age, increases if ate hay, etc. 30

Interactions in a regression • Normally, outcome = ax 1 + bx 2 + c + … • Interactions account for situations when two variables are not simply additive. Instead, their interaction impacts the outcome – e.g., Maybe brown ponies, and only brown ponies, get a larger benefit from eating pop-tarts before a race • Outcome = ax 1 + bx 2 + c + d(x 1 x 2 ) + … 31

Example logistic regression output Factor Coef. Exp(coef) SE p -value number of digits -0.343 0.709 0.009 < 0.001 number of lowercase -0.355 0.701 0.008 < 0.001 number of uppercase -0.783 0.457 0.028 < 0.001 number of symbols -0.582 0.559 0.037 < 0.001 digits in middle -0.714 0.490 0.040 < 0.001 < 0.001 digits spread out -1.624 0.197 0.051 digits at beginning -0.256 0.774 0.066 < 0.001 uppercase in middle -0.168 0.845 0.105 0.108 † uppercase spread out 0.055 1.057 0.114 0.629 † uppercase at beginning 0.631 1.879 0.105 < 0.001 symbols in middle -0.844 0.430 0.038 < 0.001 symbols spread out -1.217 0.296 0.085 < 0.001 < 0.001 symbols at beginning -0.287 0.751 0.070 gender (male) -4.4 E -4 1.000 0.023 0.985 † birth year 0.005 1.005 0.001 < 0.001 engineering -0.140 0.870 0.042 < 0.001 humanities -0.078 0.925 0.049 0.108 † 0.576 † public policy 0.029 1.029 0.051 science -0.161 0.851 0.055 0.003 0.154 † other -0.066 0.936 0.046 computer science -0.195 0.823 0.047 < 0.001 business 0.167 1.182 0.049 < 0.001 32