Power, Sample Size, and the FDR Peter Dalgaard Department of Biostatistics University of Copenhagen Center for Bioinformatics, Univ.Copenhagen, June 2005
Sample Size “How many observations do we need?” Depends on • Design • Standard error of measurements • Effect size • How sure you want to be of finding it
Sample Size “How many observations do we need?” Depends on • Design • Standard error of measurements • Effect size • How sure you want to be of finding it
Sample Size “How many observations do we need?” Depends on • Design • Standard error of measurements • Effect size • How sure you want to be of finding it
Sample Size “How many observations do we need?” Depends on • Design • Standard error of measurements • Effect size • How sure you want to be of finding it
Sample Size “How many observations do we need?” Depends on • Design • Standard error of measurements • Effect size • How sure you want to be of finding it
Reminders (Continuous data) One-sample (or paired, differences): � SEM = s × 1 / n Significance if | ¯ x − µ 0 | > t . 975 ( DF ) SEM Two-sample: � SEDM = s × 1 / n 1 + 1 / n 2 | ¯ x 1 − ¯ x 2 | SEDM > t . 975 ( DF ) t . 975 ( DF ) ≈ 2. Notice that SE(D)M decreases with n .
Reminders (Continuous data) One-sample (or paired, differences): � SEM = s × 1 / n Significance if | ¯ x − µ 0 | > t . 975 ( DF ) SEM Two-sample: � SEDM = s × 1 / n 1 + 1 / n 2 | ¯ x 1 − ¯ x 2 | SEDM > t . 975 ( DF ) t . 975 ( DF ) ≈ 2. Notice that SE(D)M decreases with n .
Reminders (Continuous data) One-sample (or paired, differences): � SEM = s × 1 / n Significance if | ¯ x − µ 0 | > t . 975 ( DF ) SEM Two-sample: � SEDM = s × 1 / n 1 + 1 / n 2 | ¯ x 1 − ¯ x 2 | SEDM > t . 975 ( DF ) t . 975 ( DF ) ≈ 2. Notice that SE(D)M decreases with n .
Reminders (Continuous data) One-sample (or paired, differences): � SEM = s × 1 / n Significance if | ¯ x − µ 0 | > t . 975 ( DF ) SEM Two-sample: � SEDM = s × 1 / n 1 + 1 / n 2 | ¯ x 1 − ¯ x 2 | SEDM > t . 975 ( DF ) t . 975 ( DF ) ≈ 2. Notice that SE(D)M decreases with n .
Reminders (Continuous data) One-sample (or paired, differences): � SEM = s × 1 / n Significance if | ¯ x − µ 0 | > t . 975 ( DF ) SEM Two-sample: � SEDM = s × 1 / n 1 + 1 / n 2 | ¯ x 1 − ¯ x 2 | SEDM > t . 975 ( DF ) t . 975 ( DF ) ≈ 2. Notice that SE(D)M decreases with n .
Variation of Observations and Means dnorm(x, sd = sqrt(1/20)) 1.5 1.0 0.5 0.0 −3 −2 −1 0 1 2 3 x
t Test 0.4 0.3 dt(t, 38) 0.2 0.1 0.0 −3 −2 −1 0 1 2 3 t • If there is no (true) difference then there is little chance of getting an observation in the tails • If there is a difference, then the center of the distribution is shifted.
t Test 0.4 0.3 dt(t, 38) 0.2 0.1 0.0 −3 −2 −1 0 1 2 3 t • If there is no (true) difference then there is little chance of getting an observation in the tails • If there is a difference, then the center of the distribution is shifted.
Type I and Type II Errors A test of a hypothesis can go wrong in two ways: Type I error: Rejecting a true null hypothesis Type II error: Accepting a false null hypothesis Error probabilities: α resp. β α : Significance level (0.05, e.g.) 1 − β : Power – probability of detecting difference Notice that the power depends on the effect size as well as on the number of observations and significance level.
Type I and Type II Errors A test of a hypothesis can go wrong in two ways: Type I error: Rejecting a true null hypothesis Type II error: Accepting a false null hypothesis Error probabilities: α resp. β α : Significance level (0.05, e.g.) 1 − β : Power – probability of detecting difference Notice that the power depends on the effect size as well as on the number of observations and significance level.
Type I and Type II Errors A test of a hypothesis can go wrong in two ways: Type I error: Rejecting a true null hypothesis Type II error: Accepting a false null hypothesis Error probabilities: α resp. β α : Significance level (0.05, e.g.) 1 − β : Power – probability of detecting difference Notice that the power depends on the effect size as well as on the number of observations and significance level.
Type I and Type II Errors A test of a hypothesis can go wrong in two ways: Type I error: Rejecting a true null hypothesis Type II error: Accepting a false null hypothesis Error probabilities: α resp. β α : Significance level (0.05, e.g.) 1 − β : Power – probability of detecting difference Notice that the power depends on the effect size as well as on the number of observations and significance level.
Calculating n – Preliminaries • (First consider one-sample case) • Wish to find difference of δ = µ − µ 0 (“clinically relevant difference”), • Naive guess: n should satisfy δ = 2 × SEM? • But the observed difference is not precisely δ . It is smaller with 50% probability, and then it wouldn’t be significant. • We need to make SEM so small that there is a high probability of getting a significant result
Calculating n – Preliminaries • (First consider one-sample case) • Wish to find difference of δ = µ − µ 0 (“clinically relevant difference”), • Naive guess: n should satisfy δ = 2 × SEM? • But the observed difference is not precisely δ . It is smaller with 50% probability, and then it wouldn’t be significant. • We need to make SEM so small that there is a high probability of getting a significant result
Calculating n – Preliminaries • (First consider one-sample case) • Wish to find difference of δ = µ − µ 0 (“clinically relevant difference”), • Naive guess: n should satisfy δ = 2 × SEM? • But the observed difference is not precisely δ . It is smaller with 50% probability, and then it wouldn’t be significant. • We need to make SEM so small that there is a high probability of getting a significant result
Calculating n – Preliminaries • (First consider one-sample case) • Wish to find difference of δ = µ − µ 0 (“clinically relevant difference”), • Naive guess: n should satisfy δ = 2 × SEM? • But the observed difference is not precisely δ . It is smaller with 50% probability, and then it wouldn’t be significant. • We need to make SEM so small that there is a high probability of getting a significant result
Calculating n – Preliminaries • (First consider one-sample case) • Wish to find difference of δ = µ − µ 0 (“clinically relevant difference”), • Naive guess: n should satisfy δ = 2 × SEM? • But the observed difference is not precisely δ . It is smaller with 50% probability, and then it wouldn’t be significant. • We need to make SEM so small that there is a high probability of getting a significant result
Power, Sketch of Principle 0.4 0.3 dnorm(x) 0.2 0.1 0.0 −2 0 2 4 6 x (x axis in units of SEM)
Size of SEM relative to δ (Notice: These formulas assume known SD. Watch out if n is very small. More accurate formulas in R’s power.t.test ) z p quantiles in normal distribution, z 0 . 975 = 1 . 96, etc. Two-tailed test, α = 0 . 05, power 1 − β = 0 . 90 δ = ( 1 . 96 + k ) × SEM k is distance between middle and right peak in slide 8. Find k so that there is a probability of 0.90 of observing a difference of at least 1 . 96 × SEM. k = − z 0 . 10 = z 0 . 90 z 0 . 90 = 1 . 28, so δ = 3 . 24 × SEM
Size of SEM relative to δ (Notice: These formulas assume known SD. Watch out if n is very small. More accurate formulas in R’s power.t.test ) z p quantiles in normal distribution, z 0 . 975 = 1 . 96, etc. Two-tailed test, α = 0 . 05, power 1 − β = 0 . 90 δ = ( 1 . 96 + k ) × SEM k is distance between middle and right peak in slide 8. Find k so that there is a probability of 0.90 of observing a difference of at least 1 . 96 × SEM. k = − z 0 . 10 = z 0 . 90 z 0 . 90 = 1 . 28, so δ = 3 . 24 × SEM
Size of SEM relative to δ (Notice: These formulas assume known SD. Watch out if n is very small. More accurate formulas in R’s power.t.test ) z p quantiles in normal distribution, z 0 . 975 = 1 . 96, etc. Two-tailed test, α = 0 . 05, power 1 − β = 0 . 90 δ = ( 1 . 96 + k ) × SEM k is distance between middle and right peak in slide 8. Find k so that there is a probability of 0.90 of observing a difference of at least 1 . 96 × SEM. k = − z 0 . 10 = z 0 . 90 z 0 . 90 = 1 . 28, so δ = 3 . 24 × SEM
Calculating n Just insert SEM = σ/ √ n in δ = 3 . 24 × SEM and solve for n : n = ( 3 . 24 × σ/δ ) 2 (for two-sided test at level α = 0 . 05, with power 1 − β = 0 . 90) General formula for arbitrary α and β : n = (( z 1 − α/ 2 + z 1 − β ) × ( σ/δ )) 2 = ( σ/δ ) 2 × f ( α, β ) next slide
Calculating n Just insert SEM = σ/ √ n in δ = 3 . 24 × SEM and solve for n : n = ( 3 . 24 × σ/δ ) 2 (for two-sided test at level α = 0 . 05, with power 1 − β = 0 . 90) General formula for arbitrary α and β : n = (( z 1 − α/ 2 + z 1 − β ) × ( σ/δ )) 2 = ( σ/δ ) 2 × f ( α, β ) next slide
Recommend
More recommend
