How Surprised is the Skeptic? Simulating the Skeptic’s World STAT 113 Hypothesis Testing II The World According to the Null Hypothesis Colin Reimer Dawson October 19, 2020 1 / 17
How Surprised is the Skeptic? Simulating the Skeptic’s World How Surprised is the Skeptic? Simulating the Skeptic’s World Randomization Distribution 2 / 17
How Surprised is the Skeptic? Simulating the Skeptic’s World The Lady Tasting Tea At a 1920s party in Cambridge, UK, a lady (Dr. Muriel Bristol, a phycologist, specializing in algae) claimed she could tell whether a cup of tea had been prepared by adding milk before or after the tea was poured. A statistician, Ronald Fisher, also in attendance, proposed a blind taste test w/ 10 cups of tea, each prepared in random order. • How much success is enough to believe her? 3 / 17
How Surprised is the Skeptic? Simulating the Skeptic’s World The Null Hypothesis • R.A. Fisher: Formulate the “most boring” hypothesis about the world/ process / population (“nothing to see here; moving along”) • Try to measure how surprising the data would have been if the “boring” thing were true. • Fisher called this boring “antihypothesis” the null hypothesis (abbreviated as H 0 ) 4 / 17
How Surprised is the Skeptic? Simulating the Skeptic’s World The Alternative Hypothesis • Jerzy Neyman and Egon Pearson added the idea of a specific alternative hypothesis to this formulation • The “alternative” is usually the one that you started with H 0 : the new drug works no better than the old one H 1 : the new drug works better than the old one H 0 : there is no relationship between bill and tip percent H 1 : there is some relationship between bill and tip percent 5 / 17
How Surprised is the Skeptic? Simulating the Skeptic’s World Hypotheses Are About Parameters • We know what’s true about our dataset (the sample ) • Our hypotheses propose possibilities involving the wider context ( population / process ) • Important: When formulating statistical hypotheses, they will always be about the population/process/phenomenon • Correct H 1 : A majority of all U.S. registered voters plan to vote for Biden in November • Incorrect H 1 : A majority of the registered voters in the poll plan to vote for Biden in November 6 / 17
How Surprised is the Skeptic? Simulating the Skeptic’s World Self-Check: Null and Alternative Hypotheses For the following research claims and datasets, identify (a) the relevant parameter(s) and the context where it applies (b) the statistic(s) that we can use to estimate the parameter(s), (c) the null hypothesis ( H 0 ), and (d) the alternative hypothesis ( H 1 ) 1. Claim : There is a positive linear association between pH and mercury in Florida lakes. 7 / 17
How Surprised is the Skeptic? Simulating the Skeptic’s World Self-Check: Null and Alternative Hypotheses For the following research claims and datasets, identify (a) the relevant parameter(s) and the context where it applies (b) the statistic(s) that we can use to estimate the parameter(s), (c) the null hypothesis ( H 0 ), and (d) the alternative hypothesis ( H 1 ) 1. Claim : There is a positive linear association between pH and mercury in Florida lakes. • Data : We measure pH and mercury levels in 50 random lakes in Florida. 7 / 17
How Surprised is the Skeptic? Simulating the Skeptic’s World Self-Check: Null and Alternative Hypotheses For the following research claims and datasets, identify (a) the relevant parameter(s) and the context where it applies (b) the statistic(s) that we can use to estimate the parameter(s), (c) the null hypothesis ( H 0 ), and (d) the alternative hypothesis ( H 1 ) 1. Claim : There is a positive linear association between pH and mercury in Florida lakes. • Data : We measure pH and mercury levels in 50 random lakes in Florida. 2. Claim : Dr. Bristol can tell the difference between milk-first and tea-first preparations better than a coin flip 7 / 17
How Surprised is the Skeptic? Simulating the Skeptic’s World Self-Check: Null and Alternative Hypotheses For the following research claims and datasets, identify (a) the relevant parameter(s) and the context where it applies (b) the statistic(s) that we can use to estimate the parameter(s), (c) the null hypothesis ( H 0 ), and (d) the alternative hypothesis ( H 1 ) 1. Claim : There is a positive linear association between pH and mercury in Florida lakes. • Data : We measure pH and mercury levels in 50 random lakes in Florida. 2. Claim : Dr. Bristol can tell the difference between milk-first and tea-first preparations better than a coin flip • Data : She tastes 10 cups in a blind taste test. 7 / 17
How Surprised is the Skeptic? Simulating the Skeptic’s World Self-Check: Null and Alternative Hypotheses For the following research claims and datasets, identify (a) the relevant parameter(s) and the context where it applies (b) the statistic(s) that we can use to estimate the parameter(s), (c) the null hypothesis ( H 0 ), and (d) the alternative hypothesis ( H 1 ) 1. Claim : There is a positive linear association between pH and mercury in Florida lakes. • Data : We measure pH and mercury levels in 50 random lakes in Florida. 2. Claim : Dr. Bristol can tell the difference between milk-first and tea-first preparations better than a coin flip • Data : She tastes 10 cups in a blind taste test. 3. Claim : Lab mice eat more on average if the room is light at meal time than if it is dark at meal time. 7 / 17
How Surprised is the Skeptic? Simulating the Skeptic’s World Self-Check: Null and Alternative Hypotheses For the following research claims and datasets, identify (a) the relevant parameter(s) and the context where it applies (b) the statistic(s) that we can use to estimate the parameter(s), (c) the null hypothesis ( H 0 ), and (d) the alternative hypothesis ( H 1 ) 1. Claim : There is a positive linear association between pH and mercury in Florida lakes. • Data : We measure pH and mercury levels in 50 random lakes in Florida. 2. Claim : Dr. Bristol can tell the difference between milk-first and tea-first preparations better than a coin flip • Data : She tastes 10 cups in a blind taste test. 3. Claim : Lab mice eat more on average if the room is light at meal time than if it is dark at meal time. • Data : 20 mice are randomly split into two groups. One group is fed in the light, another in the dark. Their food intake in grams is measured. 7 / 17
How Surprised is the Skeptic? Simulating the Skeptic’s World Outline How Surprised is the Skeptic? Simulating the Skeptic’s World Randomization Distribution 8 / 17
How Surprised is the Skeptic? Simulating the Skeptic’s World Logic of Testing H 0 • Imagine two observers: 9 / 17
How Surprised is the Skeptic? Simulating the Skeptic’s World Logic of Testing H 0 • Imagine two observers: • a skeptic who thinks there is nothing interesting going on (they believe H 0 ) 9 / 17
How Surprised is the Skeptic? Simulating the Skeptic’s World Logic of Testing H 0 • Imagine two observers: • a skeptic who thinks there is nothing interesting going on (they believe H 0 ) • a proponent who thinks there is something interesting there (they believe H 1 ) 9 / 17
How Surprised is the Skeptic? Simulating the Skeptic’s World Logic of Testing H 0 • Imagine two observers: • a skeptic who thinks there is nothing interesting going on (they believe H 0 ) • a proponent who thinks there is something interesting there (they believe H 1 ) • Ask which values of the statistic would surprise the proponent less than they would surprise the skeptic 9 / 17
How Surprised is the Skeptic? Simulating the Skeptic’s World Logic of Testing H 0 • Imagine two observers: • a skeptic who thinks there is nothing interesting going on (they believe H 0 ) • a proponent who thinks there is something interesting there (they believe H 1 ) • Ask which values of the statistic would surprise the proponent less than they would surprise the skeptic • Of those, sort them in descending order according to how much they favor the proponent 9 / 17
How Surprised is the Skeptic? Simulating the Skeptic’s World Logic of Testing H 0 • Imagine two observers: • a skeptic who thinks there is nothing interesting going on (they believe H 0 ) • a proponent who thinks there is something interesting there (they believe H 1 ) • Ask which values of the statistic would surprise the proponent less than they would surprise the skeptic • Of those, sort them in descending order according to how much they favor the proponent • If the data yields a statistic which is sufficiently far up that list, the skeptic will change their mind 9 / 17
How Surprised is the Skeptic? Simulating the Skeptic’s World • Dr. Bristol tastes 10 cups of tea and guesses the preparation of each. What is the statistic of interest? • Which values should suprise a proponent less than they surprise a skeptic ? • Which of those favor the proponent the most ? 10 / 17
How Surprised is the Skeptic? Simulating the Skeptic’s World The P -value • The skeptic assigns a weight to every possible value of the statistic , according to how often that value should occur in the skeptic’s model of the world 11 / 17
How Surprised is the Skeptic? Simulating the Skeptic’s World The P -value • The skeptic assigns a weight to every possible value of the statistic , according to how often that value should occur in the skeptic’s model of the world • These weights together must add up to 1 11 / 17
Recommend
More recommend
