Motivating Problems Statistical Criteria for Discovery Examples: Mass Hierarchy, CP-violation, Higgs Search Advice Statistical Quantification of Discovery in Neutrino Physics David A. van Dyk Statistics Section, Imperial College London PhyStat-nu, Fermilab, 2016 uci
Motivating Problems Statistical Criteria for Discovery Examples: Mass Hierarchy, CP-violation, Higgs Search Advice Statistical Discovery in Neutrino Physics I am a statistician, not a neutrino physicists... I collaborate with astrophysicists, solar physicists, and particle physicists on statistical methodology. First contact with neutrino physics: PhyStat- ν ...3 months ago Today: Summarize a number of statistical issues that pertain to discovery in neutrino physics ... as discussed in PhyStat- ν , Tokyo Illustrate how they play out in three examples. uci
Motivating Problems Statistical Criteria for Discovery Examples: Mass Hierarchy, CP-violation, Higgs Search Advice Outline Motivating Problems 1 Statistical Criteria for Discovery 2 Examples: Mass Hierarchy, CP-violation, Higgs Search 3 Advice 4 uci
Motivating Problems Statistical Criteria for Discovery Examples: Mass Hierarchy, CP-violation, Higgs Search Advice Outline Motivating Problems 1 Statistical Criteria for Discovery 2 Examples: Mass Hierarchy, CP-violation, Higgs Search 3 Advice 4 uci
Motivating Problems Statistical Criteria for Discovery Examples: Mass Hierarchy, CP-violation, Higgs Search Advice Motivating Problems Mass Hierarchy normal ( ∆ m 2 inverted hierarchy ( ∆ m 2 32 > 0) 32 < 0) vs | ∆ m 2 32 | well constrained, degeneracy of sign with θ 23 or δ CP . CP-violation Is there evidence to counter δ CP ∈ { 0 , π } ? Current data is limited. a 2,400 2,200 Selected diphoton sample Data 2011 and 2012 2,000 Signal + background inclusive fi t ( m H = 126.5 GeV) 1,800 Events per GeV Fourth-order polynomial 1,600 Bump Hunting (e.g., Higgs serach) s = 7 TeV, L d t = 4.8 fb –1 1,400 s = 8 TeV, L d t = 5.9 fb –1 1,200 1,000 no bump bump 800 vs 600 400 location of bump unknown 200 ATLAS internal Data – background b 100 What is the bump location if 0 –100 100 110 120 130 140 150 160 there is no bump? m γγ (GeV) uci
Motivating Problems Statistical Criteria for Discovery Examples: Mass Hierarchy, CP-violation, Higgs Search Advice Outline Motivating Problems 1 Statistical Criteria for Discovery 2 Examples: Mass Hierarchy, CP-violation, Higgs Search 3 Advice 4 uci
Motivating Problems Statistical Criteria for Discovery Examples: Mass Hierarchy, CP-violation, Higgs Search Advice Statistical Framework for Discovery Model / Hypothesis Testing H 0 : The null hypothesis (e.g., no CP-violoation, δ CP = 0) H 1 : The alternative hypothesis (e.g., CP-violation) Without further evidence, H 0 is presumed true. “Deciding” on H 1 means scientific discovery: new physics. Model Selection: No presumed model. (normal/inverted hierarchy) Appropriate Statistical Approach Depends on: Is H 0 the presumed model? are there more than 2 possible models? Is H 0 a special case of H 1 , “nested models” Parameters: (i) Unknown values under H 0 ? (ii) No “true value” under H 0 ?, (iii) Boundary concerns. Bayesian vs. Frequentist methods uci
Motivating Problems Statistical Criteria for Discovery Examples: Mass Hierarchy, CP-violation, Higgs Search Advice Statistical Criterion for Discovery The most common criterion is the p-value, � � p-value = Pr T ( y ) ≥ T ( y obs ) | H 0 T ( · ) is a Test Statistic , e.g., ∆ χ 2 or likelihood ratio statistic H 0 : NH H 1 : IH p−value T ( y obs ) T(y) uci
Motivating Problems Statistical Criteria for Discovery Examples: Mass Hierarchy, CP-violation, Higgs Search Advice Computing p-values The most common criterion is the p-value, � � p-value = Pr T ( y ) ≥ T ( y obs ) | H 0 H 0 : NH H 1 : IH p−value T ( y obs ) T(y) Requires distribution of T ( y ) under H 0 Distributions depend on unknown parameters (e.g., δ CP , θ 23 ) Standard Theory: models nested, all parameters have values under H 0 , “large” data set. ... often violated in physics Monte Carlo toys infeasible with 5 σ criterion. uci
Motivating Problems Statistical Criteria for Discovery Examples: Mass Hierarchy, CP-violation, Higgs Search Advice Misuse of P-values The most common criterion is the p-value, � � p-value = Pr with T = test statistic T ( y ) ≥ T ( y obs ) | H 0 But.... uci
Motivating Problems Statistical Criteria for Discovery Examples: Mass Hierarchy, CP-violation, Higgs Search Advice Misuse of P-values The most common criterion is the p-value, � � p-value = Pr with T = test statistic T ( y ) ≥ T ( y obs ) | H 0 But.... uci
Motivating Problems Statistical Criteria for Discovery Examples: Mass Hierarchy, CP-violation, Higgs Search Advice Misuse of P-values The most common criterion is the p-value, � � p-value = Pr with T = test statistic T ( y ) ≥ T ( y obs ) | H 0 But.... (ASA Statement on Statistical Significance and P-values) uci February 5, 2016
Motivating Problems Statistical Criteria for Discovery Examples: Mass Hierarchy, CP-violation, Higgs Search Advice The Problem with P-values The misuse of P-values: Do not measure relative likelihood of hypotheses. Large p-values do not validate H 0 . May depend on bits of H 0 that are of no interest. Single filter for publication / judging quality of research. Should be viewed as a data summary, not the summary Reviewers, Editors, and Readers want a simple black-and-white rule: p < 0 . 05 , or > 5 σ . But, statistics is about quantifying uncertainty, not expressing certainty. uci
Motivating Problems Statistical Criteria for Discovery Examples: Mass Hierarchy, CP-violation, Higgs Search Advice A Bayesian Criterion for Discovery To determine mass hierarchy, suppose we find � � p-value = Pr T ( y ) ≥ T ( y obs ) | NH = 0 . 0001 Questions Can we conclude NH is unlikely? Does Pr ( data | NH ) small imply Pr ( NH | data ) is small? Order of conditioning matters! Consider Pr ( A | B ) and Pr ( B | A ) with A: A person is a woman. B: A person is pregnant. uci
Motivating Problems Statistical Criteria for Discovery Examples: Mass Hierarchy, CP-violation, Higgs Search Advice Bayesian Methods Bayes Theorem Pr ( data | NH ) Pr ( NH ) Pr ( NH | data ) = Pr ( data | NH ) Pr ( NH ) + Pr ( data | IH ) Pr ( IH ) Bayesian methods have cleaner mathematical foundations more directly answer scientific questions ... but they depend on prior distributions Pr ( NH ) = probability of NH before seeing data. Prior distributions must also be specified for model parameters. uci
Motivating Problems Statistical Criteria for Discovery Examples: Mass Hierarchy, CP-violation, Higgs Search Advice The Problem with Priors Bayesian Criteria for Discovery: p 0 ( y ) � Bayes Factor p 1 ( y ) with p i ( y ) = = p i ( y | θ ) p i ( θ ) d θ. p 0 ( y ) π 0 π 0 Pr ( H 0 | y ) = = π 0 + π 1 ( Bayes Factor ) − 1 p 0 ( y ) π 0 + p 1 ( y ) π 1 Example: (simplified) Higgs search Likelihood: y | λ ∼ Poisson ( 10 + λ ) Test: λ = 0 vs λ > 0 0.030 0.04 marginal likelihood prior distribution 0.03 0.020 p ( λ ) 0.02 p 1 ( y ) 0.010 0.01 0.000 0.00 0 20 40 60 80 100 0 20 40 60 80 100 λ y Value of p 1 ( y ) depends on prior! uci
Motivating Problems Statistical Criteria for Discovery Examples: Mass Hierarchy, CP-violation, Higgs Search Advice Choice of Prior Matters! Bayes Factor 0.8 y ∼ Poisson ( 10 ) . H 0 : 0.6 log(Bayes Factor) y ∼ Poisson ( 10 + λ ) . H 1 : 0.4 with λ ∼ exp ( ξ ) 0.2 0.0 Observe y = 15 −0.2 log ( Bayes Factor ) −2 −1 0 1 2 log ( ξ ) Must think hard about choice of prior and report! uci
Motivating Problems Statistical Criteria for Discovery Examples: Mass Hierarchy, CP-violation, Higgs Search Advice Frequentist vs Bayesian: Does it Matter? Model Testing and Model Selection Frequency and Bayesian methods may not agree . Bayes automatically penalizes larger models (Occam’s Razor) and adjusts for trial factors / look elsewhere effect. Choice of prior distribution is often critical . Problem cases: Dimension of model parameters differ. CP-violation: H 0 : δ CP ∈ { 0 , π } vs. H 1 : δ CP / ∈ { 0 , π } . Higgs search: location and intensity of bump above bkgd. Anti-conservative: p-value ≪ Pr ( H 0 | y ) . Remember: p-value and Pr ( H 0 | y ) quantify different things! Interpreting p-value as Pr ( H 0 | y ) may significantly overstate evidence for new physics. uci
Motivating Problems Statistical Criteria for Discovery Examples: Mass Hierarchy, CP-violation, Higgs Search Advice Example: Searching for a bump above background. E.g., in toy version of Higgs search with known mass... 1.0 Bound on P ( H 0 | Y , µ ) p−value / P ( H 0 | Y ) 0.8 p−value P ( H 0 | Y , µ ) 0.6 0.4 0.2 0.0 250 300 350 400 450 500 count .... but researchers interpret p-value as Pr ( H 0 | y ) . Solution: Report both. uci
Recommend
More recommend
