Statistics for Applications Chapter 6: Testing goodness of fit - PowerPoint PPT Presentation

Statistics for Applications Chapter 6: Testing goodness of fit 1/25

Goodness of fit tests Let X be a r.v. Given i.i.d copies of X we want to answer the following types of questions: ◮ Does X have distribution N (0 , 1) ? (Cf. Student’s T distribution) ◮ Does X have distribution U ([0 , 1]) ? (Cf p-value under H 0 ) ◮ Does X have PMF p 1 = 0 . 3 , p 2 = 0 . 5 , p 3 = 0 . 2 These are all goodness of fit tests: we want to know if the hypothesized distribution is a good fit for the data. Key characteristic of GoF tests: no parametric modeling. 2/25

Cdf and empirical cdf (1) Let X 1 , . . . , X n be i.i.d. real random variables. Recall the cdf of X 1 is defined as: F ( t ) = I P[ X 1 ≤ t ] , ∀ t ∈ I R . It completely characterizes the distribution of X 1 . Definition The cdf of the sample X 1 , . . . , X n is defined as: empirical n 1 L 1 { X i ≤ t } F n ( t ) = n i =1 # { i = 1 , . . . , n : X i ≤ t } = , ∀ t ∈ I R . n 3/25

Cdf and empirical cdf (2) By the LLN, for all t ∈ I R , a.s. F n ( t ) − − − → F ( t ) . n →∞ Theorem ( Fundamental theorem of Glivenko-Cantelli statistics ) a.s. sup | F n ( t ) − F ( t ) | − − − → 0 . n →∞ t ∈ I R 4/25

Cdf and empirical cdf (3) By the CLT, for all t ∈ I R , √ ( d ) ( ) n ( F n ( t ) − F ( t )) − − − → N 0 , F ( t ) (1 − F ( t )) . n →∞ Donsker’s Theorem If F is continuous, then √ ( d ) n sup | F n ( t ) − F ( t ) | − − − → sup | B ( t ) | , n →∞ 0 ≤ t ≤ 1 t ∈ I R B is where a Brownian bridge on [0 , 1] . 5/25

Kolmogorov-Smirnov test (1) ◮ Let X 1 , . . . , X n be i.i.d. real random variables with unknown F 0 be continuous cdf. cdf F and let a ◮ Consider the two hypotheses: F 0 F 0 H 0 : F = v.s. H 1 : F = . ◮ Let F n be the empirical cdf of the sample X 1 , . . . , X n . F 0 , F n ( t ) ≈ F 0 ( t ) , ◮ If ∈ [0 , 1] . F = then for all t 6/25

Kolmogorov-Smirnov test (2) √ n F n ( t ) − F 0 ( t ) ◮ Let T n = sup . t ∈ I R ( d ) ◮ By Donsker’s theorem, if H 0 is true, then T n − − − → Z , n →∞ where Z has a known distribution (supremum of a Brownian bridge). ◮ KS test with asymptotic level α : δ KS = 1 { T n > q α } , α where q α is the (1 − α ) -quantile of Z (obtained in tables). ◮ p-value of KS test: I P[ Z > T n | T n ] . 7/25

Kolmogorov-Smirnov test (3) Remarks: ◮ In practice, how to compute T n ? ◮ F 0 is non decreasing, F n is piecewise constant, with jumps at t i = X i , i = 1 , . . . , n . ◮ Let X (1) ≤ X (2) ≤ . . . ≤ X ( n ) be the reordered sample. ◮ The expression for T n reduces to the following practical formula: √ i − 1 i { } − F 0 ( X ( i ) ) , − F 0 ( X ( i ) ) T n = n max max . n n i =1 ,...,n 8/25

Kolmogorov-Smirnov test (4) ◮ T n is called a pivotal statistic : If H 0 is true, the distribution of T n does not depend on the distribution of the X i ’s and it is easy to reproduce it in simulations. F 0 ( X i ) , i ◮ Indeed, let U i = = 1 , . . . , n and let G n be the empirical cdf of U 1 , . . . , U n . i.i.d. U 1 , . . . , U n ∼ U ([0 . 1]) ◮ If H 0 is true, then √ n | G n ( x ) − x | . and T n = sup 0 ≤ x ≤ 1 9/25

Kolmogorov-Smirnov test (5) ◮ For some large integer M : T 1 , . . . , T M of ◮ Simulate M i.i.d. copies T n ; n n ( n ) (1 − α ) -quantile ◮ Estimate the q α of T n by taking the sample ( n,M ) 1 , . . . , T (1 − α ) -quantile M q ˆ of T n . α n ◮ Test with approximate level α : ( n,M ) δ α = 1 { T n > q ˆ } . α ◮ Approximate p-value of this test: j # { j = 1 , . . . , M : T n > T n } p-value ≈ . M 10/25

Kolmogorov-Smirnov test (6) These quantiles are often precomputed in a table. 11/25

Other goodness of fit tests We want to measure the distance between two functions: F n ( t ) and F ( t ) . There are other ways, leading to other tests: ◮ Kolmogorov-Smirnov: d ( F n , F ) = sup | F n ( t ) − F ( t ) | t ∈ I R ◮ Cram´ er-Von Mises: � d 2 ( F n , F ) = [ F n ( t ) − F ( t )] 2 dt I R ◮ Anderson-Darling: [ F n ( t ) − F ( t )] 2 � d 2 ( F n , F ) = dt F ( t )(1 − F ( t )) I R 12/25

Composite goodness of fit tests What if I want to test: ”Does X have Gaussian distribution?” but I don’t know the parameters? Simple idea: plug-in sup F n ( t ) − Φ ˆ σ 2 ( t ) µ, ˆ t ∈ I R where ¯ σ 2 S 2 µ ˆ = X n , ˆ = n ˆ 2 ) . and Φ ˆ σ 2 ( t ) is the cdf of N (ˆ µ, σ µ, ˆ In this case Donsker’s theorem is longer valid . This is a no common and serious mistake! 13/25

Kolmogorov-Lilliefors test (1) Instead, we compute the quantiles for the test statistic: sup F n ( t ) − Φ ˆ σ 2 ( t ) µ, ˆ t ∈ I R They do not depend on unknown parameters! This is the Kolmogorov-Lilliefors test. 14/25

Kolmogorov-Lilliefors test (2) These quantiles are often precomputed in a table. 15/25

Quantile-Quantile (QQ) plots (1) ◮ Provide a visual way to perform GoF tests ◮ Not formal test but quick and easy check to see if a distribution is plausible. ◮ Main idea: we want to check visually if the plot of F n is close F − 1 is to that of F or equivalently if the plot of close to that n F − 1 of . ◮ More convenient to check if the points − 1 − 1 ) 1 1 ) ( 2 2 ) n n F − 1 ( ) , F − 1 ( ) , F − 1 ( ) , F − 1 ( ) , . . . , F − 1 ( ) , F − 1 ( ( ( ) n n n n n n n n n are near the line y = x . ◮ F n is not technically invertible but we define F − 1 ( i/n ) = X ( i ) , n the i th largest observation. 16/25

χ 2 goodness-of-fit test, finite case (1) ◮ Let X 1 , . . . , X n be i.i.d. random variables on some finite space E = { a 1 , . . . , a K } , with some probability measure I P . ◮ Let (I P θ ) θ ∈ Θ be a parametric family of probability distributions on E . ◮ Example: On E = { 1 , . . . , K } , consider the family of binomial distributions ( Bin ( K, p )) p ∈ (0 , 1) . ∈ Θ , ◮ For j = 1 , . . . , K and θ set p j ( θ ) = I P θ [ Y = a j ] , where Y ∼ I P θ and p j = I P[ X 1 = a j ] . 19/25

χ 2 goodness-of-fit test, finite case (2) ◮ Consider the two hypotheses: ∈ (I ∈ H 0 : I P P θ ) v.s. H 1 : I P / (I P θ ) . θ ∈ Θ θ ∈ Θ ◮ Testing H 0 means testing whether the statistical model ( ) E, (I P θ ) θ ∈ Θ fits the data (e.g., whether the data are indeed from a binomial distribution). ◮ H 0 is equivalent to: p j ( θ ) , ∀ j = 1 , . ∈ Θ . p j = . . , K, for some θ 20/25

χ 2 goodness-of-fit test, finite case (3) ˆ be ◮ Let θ the MLE of θ when assuming H 0 is true. ◮ Let n # { i a j } 1 : X i = L p ˆ j = 1 { X i = a j } = , j = 1 , . . . , K . n n i =1 ˆ) are ◮ Idea: If H 0 is true, then p j = p j ( θ ) so both p ˆ j and p j ( θ ˆ) , ∀ j = 1 , . . . , K . good estimators or p j . Hence, p ˆ j ≈ p j ( θ � 2 � ˆ) K p ˆ j − p j ( θ L ◮ Define the test statistic: T n = n . p j (ˆ θ ) j =1 21/25

χ 2 goodness-of-fit test, finite case (4) ◮ Under some technical assumptions, if H 0 is true, then ( d ) → χ 2 T n − − − K − d − 1 , n →∞ R d and where d is the size of the parameter θ ( Θ ⊆ I − 1 ). d < K ◮ Test ∈ (0 , 1) : with asymptotic level α δ α = 1 { T n > q α } , χ 2 (1 − α ) -quantile where q α is the of K − d − 1 . ∼ χ 2 ◮ p-value: I P[ Z > T n | T n ] , where Z and Z ⊥ ⊥ T n . K − d − 1 22/25

χ 2 goodness-of-fit test, infinite case (1) ◮ If E is infinite (e.g. E = I N , E = I R , ...): ◮ Partition E into K disjoint bins: E = A 1 ∪ . . . ∪ A K . ◮ Define, for θ ∈ Θ and j = 1 , . . . , K : ◮ p j ( θ ) = I P θ [ Y ∈ A j ] , for Y ∼ I P θ , ◮ p j = I P[ X 1 ∈ A j ] , n 1 # { i : X i ∈ A j } L ◮ p ˆ j = 1 { X i ∈ A j } = , n n i =1 ˆ : same ◮ θ as in the previous case. 23/25

χ 2 goodness-of-fit test, infinite case (2) 2 � � ˆ) p ˆ j − p j ( θ K L ◮ As previously, let T n = n . ˆ) p j ( θ j =1 ◮ Under some technical assumptions, if H 0 is true, then ( d ) → χ 2 T n − − − K − d − 1 , n →∞ R d and where d is the size of the parameter θ ( Θ ⊆ I d < K − 1 ). ◮ Test with asymptotic level α ∈ (0 , 1) : δ α = 1 { T n > q α } , χ 2 where q α is the (1 − α ) -quantile of K − d − 1 . 24/25

χ 2 goodness-of-fit test, infinite case (3) ◮ Practical issues: ◮ Choice of K ? ◮ Choice of the bins A 1 , . . . , A K ? ◮ Computation of p j ( θ ) ? ∈ ( Poiss ( λ )) λ> 0 . ◮ Example 1: Let E = I N and H 0 : I P ◮ If one expects λ to be no larger than some λ max , one can { 0 } , A 2 = { 1 } , . . . , A K − 1 = { K − 2 } , A K = choose A 1 = { K − 1 , K, K + 1 , . . . } , with K large enough such that p K ( λ max ) ≈ 0 . 25/25

MIT OpenCourseWare https://ocw.mit.edu 18.650 / 18.6501 Statistics for Applications Fall 2016 For information about citing these materials or our Terms of Use, visit: https://ocw.mit.edu/terms.