Optimal and maximin procedures for multiple testing problems Saharon Rosset Tel Aviv University With: Ruth Heller, Amichai Painsky, Ehud Aharoni. arxiv.org/abs/1804.10256 arxiv.org/abs/1902.00892 Saharon RossetTel Aviv University Optimal multiple testing
Two normal means, FWER control Bonferroni-Holm Closed testing using Stouffer 49 0.20 0.20 0.15 0.15 0.10 0.10 u 1 u 1 0.05 0.05 0.00 0.00 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 u 2 u 2 Optimal multiple test (OMT) for two false nulls: θ 0 = − 0 . 5 θ 0 = − 1 θ 0 = − 2 0.20 0.20 0.20 0.15 0.15 0.15 0.10 0.10 0.10 u 1 u 1 u 1 0.05 0.05 0.05 0.00 0.00 0.00 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 u 2 u 2 u 2 Saharon RossetTel Aviv University Optimal multiple testing
Hypothesis testing basics Given some data X we want to test: H 0 : X ∼ F 0 H A : X ∼ F A Assume F 0 and F A have density f 0 , f a respectively, then Neyman-Pearson (NP) Lemma says that a most powerful (MP) test rejects H 0 at x iff f a ( x ) / f 0 ( x ) ≥ c . Different formulation in terms of p-value : We transform using the distribution of the likelihood ratio to get: H 0 : U = H ( X ) ∼ U ( 0 , 1 ) H A : U ∼ G and G has density g ( u ) that is a decreasing function. Now NP says MP test at level α rejects H 0 iff U ≤ α . Saharon RossetTel Aviv University Optimal multiple testing
Hypothesis testing basics Given some data X we want to test: H 0 : X ∼ F 0 H A : X ∼ F A Assume F 0 and F A have density f 0 , f a respectively, then Neyman-Pearson (NP) Lemma says that a most powerful (MP) test rejects H 0 at x iff f a ( x ) / f 0 ( x ) ≥ c . Different formulation in terms of p-value : We transform using the distribution of the likelihood ratio to get: H 0 : U = H ( X ) ∼ U ( 0 , 1 ) H A : U ∼ G and G has density g ( u ) that is a decreasing function. Now NP says MP test at level α rejects H 0 iff U ≤ α . Saharon RossetTel Aviv University Optimal multiple testing
Most powerful tests as an optimization problem We can think of the MP problem as an optimization problem on an infinite set of variables: � 1 max D ( u ) g ( u ) du D :[ 0 , 1 ] →{ 0 , 1 } 0 � 1 s.t. D ( u ) du ≤ α 0 This (integer, infinite) problem happens to have the simple solution structure implied by the NP Lemma (basically a continuous knapsack problem), because it has just one constraint. Saharon RossetTel Aviv University Optimal multiple testing
Most powerful tests as an optimization problem We can think of the MP problem as an optimization problem on an infinite set of variables: � 1 max D ( u ) g ( u ) du D :[ 0 , 1 ] →{ 0 , 1 } 0 � 1 s.t. D ( u ) du ≤ α 0 This (integer, infinite) problem happens to have the simple solution structure implied by the NP Lemma (basically a continuous knapsack problem), because it has just one constraint. Saharon RossetTel Aviv University Optimal multiple testing
Moving to multiple testing setup In a multiple testing problem, we are given K pairs of hypotheses: H 0 k : U k ∼ U ( 0 , 1 ) H Ak : U k ∼ G (assume for now all alternatives are the same). In the paper we deal with (exchangeable) dependence, here we also assume U j , U k are independent for j � = k . We seek to design good tests that give high power while controlling type-I error (level) . Saharon RossetTel Aviv University Optimal multiple testing
Moving to multiple testing setup In a multiple testing problem, we are given K pairs of hypotheses: H 0 k : U k ∼ U ( 0 , 1 ) H Ak : U k ∼ G (assume for now all alternatives are the same). In the paper we deal with (exchangeable) dependence, here we also assume U j , U k are independent for j � = k . We seek to design good tests that give high power while controlling type-I error (level) . Saharon RossetTel Aviv University Optimal multiple testing
Moving to multiple testing setup In a multiple testing problem, we are given K pairs of hypotheses: H 0 k : U k ∼ U ( 0 , 1 ) H Ak : U k ∼ G (assume for now all alternatives are the same). In the paper we deal with (exchangeable) dependence, here we also assume U j , U k are independent for j � = k . We seek to design good tests that give high power while controlling type-I error (level) . Saharon RossetTel Aviv University Optimal multiple testing
Some notation h ∈ { 0 , 1 } K is the true state of all hypotheses: h k = 1 ⇔ H Ak holds . D : [ 0 , 1 ] K → { 0 , 1 } K is the decision function: Rejects H 0 k at u ∈ [ 0 , 1 ] K ⇔ D k ( u ) = 1. R ( D )( u ) = � K k = 1 D ( u ) is the number of rejected nulls at u according to D . V ( D )( u ) = � K k = 1 , h k = 0 D ( u ) is the number of type-I errors at u according to D . We only consider symmetric D functions: σ ( D ( u )) = D ( σ ( u )) for any permutation σ . Saharon RossetTel Aviv University Optimal multiple testing
Some notation h ∈ { 0 , 1 } K is the true state of all hypotheses: h k = 1 ⇔ H Ak holds . D : [ 0 , 1 ] K → { 0 , 1 } K is the decision function: Rejects H 0 k at u ∈ [ 0 , 1 ] K ⇔ D k ( u ) = 1. R ( D )( u ) = � K k = 1 D ( u ) is the number of rejected nulls at u according to D . V ( D )( u ) = � K k = 1 , h k = 0 D ( u ) is the number of type-I errors at u according to D . We only consider symmetric D functions: σ ( D ( u )) = D ( σ ( u )) for any permutation σ . Saharon RossetTel Aviv University Optimal multiple testing
Some notation h ∈ { 0 , 1 } K is the true state of all hypotheses: h k = 1 ⇔ H Ak holds . D : [ 0 , 1 ] K → { 0 , 1 } K is the decision function: Rejects H 0 k at u ∈ [ 0 , 1 ] K ⇔ D k ( u ) = 1. R ( D )( u ) = � K k = 1 D ( u ) is the number of rejected nulls at u according to D . V ( D )( u ) = � K k = 1 , h k = 0 D ( u ) is the number of type-I errors at u according to D . We only consider symmetric D functions: σ ( D ( u )) = D ( σ ( u )) for any permutation σ . Saharon RossetTel Aviv University Optimal multiple testing
Some notation h ∈ { 0 , 1 } K is the true state of all hypotheses: h k = 1 ⇔ H Ak holds . D : [ 0 , 1 ] K → { 0 , 1 } K is the decision function: Rejects H 0 k at u ∈ [ 0 , 1 ] K ⇔ D k ( u ) = 1. R ( D )( u ) = � K k = 1 D ( u ) is the number of rejected nulls at u according to D . V ( D )( u ) = � K k = 1 , h k = 0 D ( u ) is the number of type-I errors at u according to D . We only consider symmetric D functions: σ ( D ( u )) = D ( σ ( u )) for any permutation σ . Saharon RossetTel Aviv University Optimal multiple testing
Generalizations of power and level The best known notions of type-I error for multiple testing: ( 1 − h ) t D ( U ) > 0 � � FWER = P ( V > 0 ) = P , R = E ( 1 − h ) t D ( U ) FDR = E V . 1 t D ( U ) Popular generalized notions of power we consider: Average power for L false nulls: � L � L Π L ( D ) = 1 � � � D l ( u ) g ( u l ) du L [ 0 , 1 ] K l = 1 l = 1 Minimal power for K false nulls: � K � K � � � Π any ( D ) = D l ( u ) > 0 g ( u l ) du [ 0 , 1 ] K I l = 1 l = 1 Saharon RossetTel Aviv University Optimal multiple testing
Generalizations of power and level The best known notions of type-I error for multiple testing: ( 1 − h ) t D ( U ) > 0 � � FWER = P ( V > 0 ) = P , R = E ( 1 − h ) t D ( U ) FDR = E V . 1 t D ( U ) Popular generalized notions of power we consider: Average power for L false nulls: � L � L Π L ( D ) = 1 � � � D l ( u ) g ( u l ) du L [ 0 , 1 ] K l = 1 l = 1 Minimal power for K false nulls: � K � K � � � Π any ( D ) = D l ( u ) > 0 g ( u l ) du [ 0 , 1 ] K I l = 1 l = 1 Saharon RossetTel Aviv University Optimal multiple testing
Resulting optimization problem for strong control max Π( D ) D :[ 0 , 1 ] K →{ 0 , 1 } K S.t. Err L ( D ) ≤ α, 0 ≤ L < K , where Π is the chosen power measure, Err is the chosen type-I error measure, and we have K and not 2 K − 1 constraints because of the symmetry “Minor” problems: D defines a continuum of variables The problem is integer The problem is not linear in D Saharon RossetTel Aviv University Optimal multiple testing
Resulting optimization problem for strong control max Π( D ) D :[ 0 , 1 ] K →{ 0 , 1 } K S.t. Err L ( D ) ≤ α, 0 ≤ L < K , where Π is the chosen power measure, Err is the chosen type-I error measure, and we have K and not 2 K − 1 constraints because of the symmetry “Minor” problems: D defines a continuum of variables The problem is integer The problem is not linear in D Saharon RossetTel Aviv University Optimal multiple testing
Monotonicity and linearity Lemma The optimal solution is always weakly monotone : u i ≤ u j ⇒ D ∗ i ( u ) ≥ D ∗ j ( u ) . Given weak monotonicity, it turns out FDR L , FWER L , Π L , Π any can all be written as linear functionals of D , for example: K � � Π any ( D ) = K ! D 1 ( u ) g ( u l ) du Q l = 1 � � � � FWER L ( D ) = L !( K − L )! D k ( u ) g ( u l ) du , Q l ∈ i k i ∈ ( K L ) , ¯ i min = k u ∈ [ 0 , 1 ] K : u 1 ≤ u 2 ≤ . . . ≤ u K � � where Q = is the ordered “corner”, and i enumerates over possible combinations of L false nulls. Saharon RossetTel Aviv University Optimal multiple testing
Recommend
More recommend
