Goodness-of-fit tests based on -entropy differences J-F Bercher 1 , - PowerPoint PPT Presentation

Goodness-of-fit tests based on φ -entropy differences J-F Bercher 1 , V. Girardin 2 , J. Lequesne 2 , P. Regnault 3 Laboratoire d’informatique Gaspard-Monge, ESIEE, Marne-la-Vallée, FRANCE 2 Laboratoire de Mathématiques N. Oresme Université de Caen – Basse Normandie, FRANCE 3 Laboratoire de Mathématiques de Reims Université de Reims Champagne-Ardenne, FRANCE MaxEnt 2014 - 25/09/14 Ph. Regnault (LMR-URCA) GOF tests via φ -entropy differences MaxEnt 2014 - 25/09/14 1 / 21

Introduction Vasicek entropy-based normality test Paradigm of GOF test via entropy differences Maximizing ( h , φ ) -entropies under moment constraints Maximum φ -entropy distributions Scale-invariant entropies A Pythagorean equality for Bregman divergence Parametric models of MaxEnt distributions... ... for Shannon entropy ... for Tsallis entropy ... for Burg entropy Goodness-of-fit tests for... ... an exponential family ... an q -exponential family Ph. Regnault (LMR-URCA) GOF tests via φ -entropy differences MaxEnt 2014 - 25/09/14 2 / 21

Introduction Vasicek entropy-based normality test Vasicek entropy-based normality test Vasicek (1976) introduced a goodness-of-fit procedure for testing the normality of uncategorical data based on the maximum entropy property N ( m , σ ) = Argmax p ∈P m ,σ S ( p ) , with ◮ N ( m , σ ) , the Gaussian distribution with mean m and variance σ 2 ; � ◮ S ( p ) = − p ( x ) log p ( x ) dx , Shannon entropy of p ; ◮ P m ,σ the set of p.d.f. with mean m and variance σ 2 . Ph. Regnault (LMR-URCA) GOF tests via φ -entropy differences MaxEnt 2014 - 25/09/14 3 / 21

Introduction Vasicek entropy-based normality test Vasicek entropy-based normality test Vasicek (1976) introduced a goodness-of-fit procedure for testing the normality of uncategorical data based on the maximum entropy property N ( m , σ ) = Argmax p ∈P m ,σ S ( p ) . Precisely, from an n -sample ( X 1 , . . . , X n ) drawn according to the p.d.f. p with finite variance , for testing H 0 : p ∈ M = {N ( m , σ ) , m ∈ R , σ > 0 } H 1 : p �∈ M , against the Vasicek test statistics is � � � S ( p ) n − S ( N ( � T n = exp m n , � σ n )) , � n � � n S ( p ) n = 1 where � 2 m ( X ( i + m ) − X ( i − m ) ) log is a consistent estimator of n i = 1 S ( p ) . Ph. Regnault (LMR-URCA) GOF tests via φ -entropy differences MaxEnt 2014 - 25/09/14 3 / 21

Introduction Paradigm of GOF test via entropy differences Paradigm of GOF test via entropy differences The main theoretical ingredients of Vasicek GOF test are : ◮ Maximum entropy property : the pdf in the null-hypothesis model M maximizes Shannon entropy under moment constraints ; ◮ Pythagorean equality : for any p ∈ P m ,σ , we have K ( p |N m ,σ ) = S ( N m ,σ ) − S ( p ); ◮ Estimation of Shannon entropy. Ph. Regnault (LMR-URCA) GOF tests via φ -entropy differences MaxEnt 2014 - 25/09/14 4 / 21

Introduction Paradigm of GOF test via entropy differences Paradigm of GOF test via entropy differences The main theoretical ingredients of Vasicek GOF test are : ◮ Maximum entropy property : the pdf in the null-hypothesis model M maximizes Shannon entropy under moment constraints ; ◮ Pythagorean equality : for any p ∈ P m ,σ , we have K ( p |N m ,σ ) = S ( N m ,σ ) − S ( p ); ◮ Estimation of Shannon entropy. Numerous authors adapted Vasicek’s procedure ◮ to various parametric models of maximum entropy distributions , where entropy stands for Shannon (overwhelming majority) or Rényi entropies ; ◮ introducing other estimators for the entropy (of both the null-hypothesis distribution and actual distribution). Ph. Regnault (LMR-URCA) GOF tests via φ -entropy differences MaxEnt 2014 - 25/09/14 4 / 21

Introduction Paradigm of GOF test via entropy differences Extending the theoretical background Parametric models for which Vasicek’s procedure-type can be developed by means of Shannon entropy maximization are well identified : exponential families ; see Lequesne’s PhD thesis. We investigate here the (informational geometric) shape and properties of parametric models for which entropy-based GOF tests may be developed, through the generalization of ◮ Maximum entropy property for φ -entropy functionals ; ◮ Pythagorean property for Bregman divergence associated to φ -entropy functionals ; ◮ φ -entropy estimation procedure adapted to the involved parametric models. Ph. Regnault (LMR-URCA) GOF tests via φ -entropy differences MaxEnt 2014 - 25/09/14 5 / 21

Maximizing ( h , φ ) -entropies under moment constraints Maximum φ -entropy distributions ( h , φ ) -entropies The ( h , φ ) -entropy of a pdf p with support S is S h ,φ ( p ) := h ( S φ ( p )) , where � S φ ( p ) = − φ ( p ( x )) dx , S with ◮ φ : R + → R a twice continuously differentiable convex function ; ◮ h : R → R a real function. ( h , φ ) – entropy h ( y ) φ ( x ) Shannon y x log x Ferreri y ( 1 + rx ) log ( 1 + rx ) / r Burg y − log x Itakura-Saito y x − log x + 1 ± ( q − 1 ) − 1 ( y − 1 ) Tsallis ∓ x q , q > 0 , q � = 1 ± ( 1 − q ) − 1 log y Rényi ∓ x q , q > 0 , q � = 1 L 2 -norm x 2 y ( 1 − 2 1 − r ) − 1 ( x r − x ) Havrda and Charvat y 1 − ( 1 + 1 / r ) x + x 1 + r / r y Basu-Harris-Hjort-Jones Ph. Regnault (LMR-URCA) GOF tests via φ -entropy differences MaxEnt 2014 - 25/09/14 6 / 21

Maximizing ( h , φ ) -entropies under moment constraints Maximum φ -entropy distributions Maximum φ -entropy distributions For increasing functions h , maximizing S h ,φ ( p ) amounts to maximizing S φ ( p ) , which is solved by : Theorem – Girardin (1997) Let M 0 = 1 , M 1 , . . . , M J linearly independent measurable functions defined on an interval S . Let m = ( 1 , m 1 , . . . , m J ) ∈ R J + 1 and p 0 ∈ P ( m , M ) , where P ( m , M ) = { p : E p ( M j ) = m j , j ∈ { 0 , . . ., J }} . If there exists (a unique) λ = ( λ 0 , . . . , λ J ) ∈ R J + 1 such that � J φ ′ ( p 0 ) = λ j M j , j = 0 then S φ ( p 0 ) ≥ S φ ( p ) , p ∈ P ( m , M ) . The converse holds if S is compact. Ph. Regnault (LMR-URCA) GOF tests via φ -entropy differences MaxEnt 2014 - 25/09/14 7 / 21

Maximizing ( h , φ ) -entropies under moment constraints Maximum φ -entropy distributions Parametric models as families of MaxEnt distributions Given a parametric family M = { p ( . ; θ ) , θ ∈ Θ ⊆ R d } of distributions supported by S , we look for ◮ φ a convex function from R + to R , ◮ M 1 , . . . , M J measurable functions from S to R with M 0 = 1 , M 1 , . . . , M J linearly independent, such that for any θ , a unique λ ∈ R J + 1 exists satisfying   � J   . p ( . ; θ ) = φ ′− 1 λ j M j j = 0 Fortunately, requiring the entropy functionals to satisfy some natural properties allows the search to be drastically restricted. Ph. Regnault (LMR-URCA) GOF tests via φ -entropy differences MaxEnt 2014 - 25/09/14 8 / 21

Maximizing ( h , φ ) -entropies under moment constraints Scale-invariant entropies Scale-invariant entropies Definitions ◮ An entropy functional S is said to be scaled-invariant if functions a and b exist, with a > 0 non-increasing such that S ( µ p µ ) = a ( µ ) S ( p ) + b ( µ ) for all µ ∈ R , where p µ ( x ) = p ( µ x ) . ◮ Two entropy functionals S and � S are said to be equivalent for maximization if � S ( p ) > � S ( p ) > S ( q ) S ( q ) . iff Ph. Regnault (LMR-URCA) GOF tests via φ -entropy differences MaxEnt 2014 - 25/09/14 9 / 21

Maximizing ( h , φ ) -entropies under moment constraints Scale-invariant entropies Scale-invariant entropies Definitions ◮ An entropy functional S is said to be scaled-invariant if functions a and b exist, with a > 0 non-increasing such that S ( µ p µ ) = a ( µ ) S ( p ) + b ( µ ) for all µ ∈ R , where p µ ( x ) = p ( µ x ) . ◮ Two entropy functionals S and � S are said to be equivalent for maximization if � S ( p ) > � S ( p ) > S ( q ) S ( q ) . iff Theorem – Kosheleva (1998) If an entropy functional is scale-invariant, then it is equivalent for maximization to one of the functionals � � � 1 − p ( x ) log p ( x ) dx , p ( x ) q dx , log p ( x ) . 1 − q S S S Ph. Regnault (LMR-URCA) GOF tests via φ -entropy differences MaxEnt 2014 - 25/09/14 9 / 21

Maximizing ( h , φ ) -entropies under moment constraints A Pythagorean equality for Bregman divergence A Pythagorean equality for Bregman divergence The Bregman divergence (or distance) D φ ( p | q ) associated to the φ -entropy of a distribution p with respect to another q with same support S is � φ ′ ( q ( x ))[ p ( x ) − q ( x )] dx . D φ ( p | q ) = S φ ( q ) − S φ ( p ) − S Entropy Associated Bregman divergence � p ( x ) log p ( x ) K ( p | q ) = Shannon q ( x ) dx , � � S � q q ( x ) q − 1 + p ( x ) q − 1 � T q ( p | q ) = ( 1 + q ) q ( x ) q dx − Tsallis p ( x ) dx , � � p ( x ) � S S q ( x ) − log p ( x ) Burg B ( p | q ) = q ( x ) − 1 . S Proposition Let p 0 ∈ P ( m , M ) satisfy φ ′ ( p 0 ) = � J j = 0 λ j M j for some λ ∈ R J + 1 . Then D φ ( p | p 0 ) = S φ ( p 0 ) − S φ ( p ) , p ∈ P ( m , M ) . Ph. Regnault (LMR-URCA) GOF tests via φ -entropy differences MaxEnt 2014 - 25/09/14 10 / 21