Multiple Comparison Procedures Cohen Chapter 13 For EDUC/PSY 6600 1
“We have to go to the deductions and the inferences,” said Lestrade, winking at me. “I find it hard enough to tackle facts, Holmes, without flying away after theories and fancies.” Inspector Lestrade to Sherlock Holmes The Boscombe Valley Mystery Cohen Chap 13 - Multiple Comparisons 2
ANOVA Omnibus: Significant F-ratio • Factor (IV) had effect on DV • Groups are not from same population • Which levels of factor differ? • Must compare and contrast means from different levels • Indicates ≥ 1 significant difference among all POSSIBLE comparisons • Simple vs . complex comparisons • Simple comparisons • Comparing 2 means, pairwise • Possible for no ‘pair’ of group means to significantly differ • Complex comparisons • Comparing combinations of > 2 means Cohen Chap 13 - Multiple Comparisons 3
Multiple Comparison Procedure • ‘Multiple comparison procedures’ used to detect simple or complex differences • Significant omnibus test NOT always necessary • Inaccurate when assumptions violated • Type II error • OKAY to conduct multiple comparisons when p -value CLOSE to significance Cohen Chap 13 - Multiple Comparisons 4
Cohen Chap 13 - Multiple Comparisons 5
Error Rates Experimentwise ( α EW ) • α = p (Type I error) p ( ≥ 1 Type I error for all • Determined in study design • Generally, α = .01, .05, or .10 comparisons) comparison error rate Relationship between α PC ( α PC ) and α EW α EW = 1 – (1 – α PC ) c α = α PC α PC = Error rate for any 1 c = Number of comparisons comparison (1 – α PC ) c = p (NOT making Type I error over c ) Cohen Chap 13 - Multiple Comparisons 6
Error rates X vs X . 1 2 X vs X . 1 3 • ANOVA with 4 groups X vs X . • F -statistic is significant 1 4 • Comparing each group with one another X vs X . • c = 6 2 3 • α PC = .05 X vs X . • α EW = _____ 2 4 • α EW when c = 10? X vs X . 3 4 • 3 Options… • Ignore α PC or α EW • Modify α PC • Modify α EW Cohen Chap 13 - Multiple Comparisons 7
Comparisons Post hoc Pre Planned (a posteriori) (a priori) Selected after data collection and Selected before data collection analysis Used in exploratory research Follow hypotheses and theory Justified conducting ANY planned Larger set of or all possible comparisons comparison (ANOVA doesn’t need to be significant) Inflated α EW : Increased p(Type I error) α EW is much smaller than alternatives α EW can slightly exceed α when planned Adjust when c is large or includes all possible comparisons?
Problems with comparisons • Decision to statistically test certain post hoc comparisons made after examining data • When only ‘most-promising’ comparisons are selected, need to correct for inflated p (Type I error) • Biased sample data often deviates from population • When all possible pairwise comparisons are conducted, p (Type I error) or α EW is same for a priori and post hoc comparisons Cohen Chap 13 - Multiple Comparisons 9
For example , a significant F -statistic is obtained: Assume 20 pairwise comparisons are possible But, in population, no significant differences exist Made a Type I error obtaining significant F -statistic However, a post hoc comparison using sample data suggests largest and smallest means differ If we had conducted 1 planned comparison 1 in 20 chance ( α = .05) of conducting this comparison and making a type I error If we had conducted all possible comparisons 100% chance ( α = 1.00) of conducting this comparison and making a type I error If researcher decides to make only 1 comparison after looking at data, between largest and smallest means, chance of type I error is still 100% All other comparisons have been made ‘in head’ and this is only one of all possible comparisons Testing largest vs. smallest means is probabilistically similar to testing all possible comparisons Cohen Chap 13 - Multiple Comparisons 10
Common techniques post hoc tests a priori tests – Fisher LSD • Multiple t -tests – Tukey HSD • Bonferroni (Dunn) – Student-Newman-Keuls • Dunn- Ŝ idák* (SNK) – Tukey-b • Holm* – Tukey-Kramer • Linear contrasts – Games-Howell – Duncan’s – Dunnett’s *adjusts α PC Italicized : not covered – REGWQ – Scheffé 11
Common techniques Many more comparison techniques available post hoc tests a priori tests Most statistical packages make no a priori / post – Fisher LSD • Multiple t -tests hoc distinction – Tukey HSD • Bonferroni (Dunn) All called post hoc (SPSS) or multiple comparisons (R) – Student-Newman-Keuls • Dunn- Ŝ idák* (SNK) In practice, most a priori comparison techniques – Tukey-b • Holm* can be used as post hoc procedures – Tukey-Kramer Called post hoc, not because they were planned after • Linear contrasts – Games-Howell doing the study per se, but because they are – Duncan’s conducted after an omnibus test – Dunnett’s *adjusts α PC Italicized : not covered – REGWQ – Scheffé 12
A Priori procedures: multiple t-tests • Homogeneity of variance • MS W (estimated pooled variance) and df W (both from ANOVA) for critical value (smaller F crit ) - - X X X X = = t 1 2 1 2 MS MS 2 MS + W W W n n n 1 2 j • Heterogeneity of variance and equal n 2 and df W with df = 2( n j - 1) for t crit • Above equation: Replace MS W with s j • Heterogeneity of variance and unequal n 2 and df W with Welch-Satterwaite • Above equation: Replace MS W with s j df for t crit Cohen Chap 13 - Multiple Comparisons 13
A Priori procedures: Bonferroni (Dunn) t-test • Bonferroni inequality • p (occurrence for set of events (additive) ≤ ∑ of probabilities for each event) • Adjusting α PC • Each comparison has p (Type I error) = α PC = .05 • α EW = .05 • α EW ≤ c* α PC • p ( ≥ 1 Type I error) can never exceed c* α PC • Conduct standard independent-samples t -tests per pair Example for 6 comparisons: α PC = .05/6 = .0083 Cohen Chap 13 - Multiple Comparisons 14
A Priori procedures: Bonferroni (Dunn) t-test t -tables lack Bonferroni-corrected critical values • Software: Exact p -values • Is exact p -value ≤ Bonferroni-corrected α -level? Example for 6 comparisons: α PC = .05/6 = .0083 More conservative : Reduced p (Type I error) Less powerful : Increased p (Type II error) Cohen Chap 13 - Multiple Comparisons 15
A Priori procedures: linear contrasts - idea • Linear combination of means: Example 1: 4 means Compare M 1 to M 2 , ignore others k = å c 1 = 1, c 2 = -1, c 3 = 0, c 4 = 0 = + + ××× + L c X c X c X c X 1 1 2 2 k k j j = + - + + = - L (1) X ( 1) X (0) X (0) X X X 1 2 3 4 1 2 = i 1 • Each group mean weighted by constant ( c ) • Products summed together • Weights selected so means of interest Example 2: Same 4 means are compared Compare M 1 , M 2 , and M 3 to M 4 c 1 = 1/3, c 2 = 1/3, c 3 = 1/3, c 4 = -1 • Sum of weights = 0 + + ( X X X ) = + + + - = - L (1/3) X (1/3) X (1/3) X ( 1) X 1 2 3 X 1 2 3 4 4 3 Cohen Chap 13 - Multiple Comparisons 16
A Priori procedures: linear contrasts - SS df for SS B = k – 1 • Each linear combination: SS Contrast df for SS Contrast = Number of ‘groups/sets’ included in contrast minus 1 Equal n s: Unequal n s: k k å å 2 ( c X ) 2 n ( c X ) j j 2 2 j j j L n L F = MS Contrast / MS W = = = j 1 = = j = j 1 SS SS Contrast æ ö æ ö Contrast k k 2 2 c c k k å å å å 2 2 c c j j MS Contrast = SS Contrast / df Contrast ç ÷ ç ÷ j j ç ÷ ç ÷ n n = = j 1 j 1 è ø è ø = = j 1 j 1 j j As df = 1, MS Contrast = SS Contrast • SS Between partitioned into k SS Contrasts MS W from omnibus ANOVA results • SS Between = SS Contrast 1 + SS Contrast 2 +…+ SS Contrast k Max # ‘legal’ contrasts = df B å 2 2 Do not need to consume all available df nL / c 2 2 MS nL L = = j = F Contrast or Use smaller α EW if # contrasts > df B å 2 æ ö MS MS c * MS 2 c k å W W j W j ç ÷ * MS ç ÷ W n è ø 17 = j 1 j
A Priori procedures: linear contrasts - example Test each Contrast (ANOVA: SS Between = 26.53, SS Within = 22.8) Mean N 9.2 5 Contrast 1: M No Noise versus M Moderate and M loud, 6.6 5 L = (-2)(9.2) + (1)(6.6) + (1)(6.2) = -18.4 + 12.8 = -5.6 SS Contrast1 = 5*(-5.6) 2 / (-2 2 + 1 2 + 1 2 ) = 156.8 / 6 = 26.13 6.2 5 df B = 2 – 1 = 1 à M S Contrast1 = 26.13/1 = 26.13 df W = 15 – 3 = 12 à MS W = 22.8/12 = 1.90 F = 26.13/1.980 = 13.75 P< .05 α =.05 & df W = 12 à F crit = 4.75 Note: SS B = SS Contrast1 + SS Contrast2 = 26.13 + 0.40 = 26.53 Cohen Chap 13 - Multiple Comparisons 18
Recommend
More recommend
