Robust mixture modeling using multivariate skew t distributions - PowerPoint PPT Presentation

Robust mixture modeling using multivariate skew t distributions Tsung-I Lin Department of Applied Mathematics and Institute of Statistics National Chung Hsing University, Taiwan August 24, 2010 ( NCHU ) 1 / 15 T.I. Lin National Chung Hsing University August 24, 2010

O UTLINE 1 Introduction 2 Preliminaries The multivariate skew t (MST) distribution 3 The multivariate skew t mixture model Model formulation and estimation 4 Example: The AIS data 5 Concluding Remarks ( NCHU ) 2 / 15 T.I. Lin National Chung Hsing University August 24, 2010

Introduction 1. I NTRODUCTION Finite mixture models have become a useful tool for modeling data that are thought to come from several different groups with varying proportions. Lin et al. (2007) proposed a novel (univariate) skew t mixture (STMIX) model, which allows for accommodation of both skewness and thick tails for making robust inferences. Drawback: limited to data with univariate outcomes. We propose a multivariate version of the STMIX (MSTMIX) model, composed of a weighed sum of g -component multivariate skew t (MST) distributions. ( NCHU ) 3 / 15 T.I. Lin National Chung Hsing University August 24, 2010

Preliminaries The multivariate skew t (MST) distribution The multivariate skew t (MST) distribution The MST distribution, Y ∼ S t p ( ξ , Σ , Λ , ν ) , can be represented by The stochastic representation of skew t distribution Y = µ + Z √ τ , Z ∼ SN p ( 0 , Σ , Λ ) , τ ∼ Γ ( ν/ 2 , ν/ 2 ) , Z ⊥ τ (1) Y | τ ∼ SN p ( µ , Σ /τ, Λ / √ τ ) Proposition 1. If τ ∼ Γ( α, β ) , then for any a ∈ R p Φ p ( a √ τ | ∆ ) � � � α � a � � � E = T p � ∆ ; 2 α . � β Integrating τ from the joint density of ( Y , τ ) yields � � ν + p � � ψ ( y | ξ , Σ , Λ , ν ) = 2 p t p ( y | ξ , Ω , ν ) T p q � ∆ ; ν + p , (2) � U + ν where q = ΛΩ − 1 ( y − ξ ) and U = ( y − ξ ) ⊤ Ω − 1 ( y − ξ ) . ( NCHU ) 4 / 15 T.I. Lin National Chung Hsing University August 24, 2010

Preliminaries The multivariate skew t (MST) distribution � 0 � � 1 � � � ρ λ 1 µ = Σ = λ = ν = 4 , , , 0 1 ρ λ 2 4 4 4 4 2 2 2 2 0 0 0 0 −2 −2 −2 −2 −4 −4 −4 −4 ( ρ, λ 1 , λ 2 ) = ( − 0.9 , 2 , 2 ) ( ρ, λ 1 , λ 2 ) = ( − 0.9 , 2 , − 2 ) ( ρ, λ 1 , λ 2 ) = ( − 0.9 , − 2 , 2 ) ( ρ, λ 1 , λ 2 ) = ( − 0.9 , − 2 , − 2 ) 4 4 4 4 2 2 2 2 0 0 0 0 −2 −2 −2 −2 −4 −4 −4 −4 ( ρ, λ 1 , λ 2 ) = ( 0 , 2 , 2 ) ( ρ, λ 1 , λ 2 ) = ( 0 , 2 , − 2 ) ( ρ, λ 1 , λ 2 ) = ( 0 , − 2 , 2 ) ( ρ, λ 1 , λ 2 ) = ( 0 , − 2 , − 2 ) 4 4 4 4 2 2 2 2 0 0 0 0 −2 −2 −2 −2 −4 −4 −4 −4 ( ρ, λ 1 , λ 2 ) = ( 0.9 , 2 , 2 ) ( ρ, λ 1 , λ 2 ) = ( 0.9 , 2 , − 2 ) ( ρ, λ 1 , λ 2 ) = ( 0.9 , − 2 , 2 ) ( ρ, λ 1 , λ 2 ) = ( 0.9 , − 2 , − 2 ) Figure 1: The scatter plots and contours and together with their histograms. ( NCHU ) 5 / 15 T.I. Lin National Chung Hsing University August 24, 2010

The multivariate skew t mixture model Model formulation and estimation The MSTMIX model The MSTMIX model g f ( y j | Θ ) = w i ψ ( y j | ξ i , Σ i , Λ i , ν i ) , � (3) i = 1 where ψ ( y j | ξ i , Σ i , Λ i , ν i ) represents the MST density, and w i ’s are the mixing probabilities satisfying � g i = 1 w i = 1. Introduce allocation variables Z j = ( Z 1 j , . . . , Z gj ) ⊤ , j = 1 , . . . , n , whose values are a set of binary variables with � 1 if Y j belongs to group i , Z ij = 0 otherwise , and satisfying � g i = 1 Z ij = 1. Denoted by Z j ∼ M ( 1 ; w 1 , . . . , w g ) . ( NCHU ) 6 / 15 T.I. Lin National Chung Hsing University August 24, 2010

The multivariate skew t mixture model Model formulation and estimation A hierarchical representation of (3) is Y j | ( γ j , τ j , Z ij = 1 ) ∼ N p ( ξ i + Λ i γ j , Σ i /τ j ) , HN p ( 0 , I p /τ j ) , γ j | ( τ j , Z ij = 1 ) ∼ τ j | ( Z ij = 1 ) ∼ Γ( ν i / 2 , ν i / 2 ) , Z j ∼ M ( 1 ; w 1 , . . . , w g ) . (4) The complete data log-likelihood function of Θ is ℓ c ( Θ | y , γ , τ , Z ) g n � log ( w i ) + ν i � ν i � ν i − 1 � � � � − log Γ 2 log | Σ i | = Z ij 2 log 2 2 i = 1 j = 1 +( ν i 2 + p − 1 ) log τ j − τ j � ( y j − ξ i − Λ i γ j ) ⊤ Σ − 1 ( y j − ξ i − Λ i γ j ) i 2 �� + ν i + γ ⊤ j γ j . ( NCHU ) 7 / 15 T.I. Lin National Chung Hsing University August 24, 2010

The multivariate skew t mixture model Model formulation and estimation Computational aspects of parameter estimation The Q function is ( k ) ) = E ( ℓ c ( Θ | y , γ , τ , Z ) | y , ˆ ( k ) ) . Q ( Θ | ˆ Θ Θ In the MCEM-based algorithm, Q -function can be approximated by M ( k ) ) = 1 ℓ c ( Θ | y , ˆ [ m ] , Z ) , Q ( Θ | ˆ ˆ � γ ∗ ( k ) τ ∗ ( k ) Θ [ m ] , ˆ (5) M m = 1 γ ∗ ( k ) γ ∗ ( k ) τ ∗ ( k ) τ ∗ ( k ) where ˆ [ m ] = { ˆ ij , m } and ˆ [ m ] = { ˆ ij , m } are independently generated by U ( k ) ν ( k ) ˆ � +ˆ ( k ) � ij , m | ( y j , Z ij = 1 ) ∼ T t p q ( k ) γ ( k + 1 ) ˆ ν ( k ) + p ; R p ˆ ij i 1 ˆ ij , ∆ , ˆ . i + ν ( k ) i p +ˆ i ij , m , y j , Z ij = 1 ) τ ( k + 1 ) γ ( k + 1 ) 2 ˆ | (ˆ ij , m ( k ) − 1 γ ( k + 1 ) q ( k ) ij ) ⊤ ˆ γ ( k + 1 ) q ( k ) U ( k ) ν ( k ) � ˆ ij ) + ˆ ν ( k ) − ˆ − ˆ (ˆ (ˆ + ˆ ∆ + 2 p i � ij , m ij , m ij i i ∼ Γ , . 2 2 ( NCHU ) 8 / 15 T.I. Lin National Chung Hsing University August 24, 2010

The multivariate skew t mixture model Model formulation and estimation The MCECM algorithm ℓ ( θ | Y o ) ( k ) ) ˆ Q ( θ | ˆ θ stopping ( 0 ) ˆ θ ℓ c ( θ | Y c ) rule ˆ MCE CM θ ( k + 1 ) ˆ θ arg max Q fix ( k ) ( k ) ˆ ˆ θ 1 θ 2 , θ 3 ( k + 1 ) ( k ) ˆ ˆ , θ 2 θ θ 1 3 ( k + 1 ) ( k + 1 ) ˆ ˆ , θ 3 θ θ 1 2 ( NCHU ) 9 / 15 T.I. Lin National Chung Hsing University August 24, 2010

The multivariate skew t mixture model Model formulation and estimation CM-steps: n w ( k + 1 ) n − 1 � z ( k ) ˆ ˆ = i ij j = 1 ( k ) y j − ˆ � n τ ( k ) � n η ( k ) j = 1 ˆ Λ j = 1 ˆ ( k + 1 ) i ˆ ij ij = ξ i τ ( k ) � n j = 1 ˆ ij ( k ) − 1 ( k ) − 1 ( k + 1 ) �� ˆ ( k ) ( k ) � ˆ diag ⊙ ˆ B � − 1 � ˆ ⊙ ˆ B 1 p � Λ = Σ Σ i i 1 i i 2 i n 1 ( k + 1 ) � ( k + 1 ) ( k + 1 ) ( y j − ˆ )( y j − ˆ ˆ � τ ( k ) ) ⊤ Σ = ˆ ξ ξ i i i ij � n z ( k ) j = 1 ˆ ij j = 1 ( k ) ⊤ ( k + 1 ) ( k ) ( k + 1 ) ( k + 1 ) ( k ) ( k + 1 ) � B B B +ˆ ˆ 1 i ˆ − ˆ ˆ 2 i − ˆ ˆ Λ Λ Λ Λ i i i 2 i i ν ( k + 1 ) Obtain ˆ as the solution of i n � ν i � ν i 1 � � + 1 − DG � κ ( k ) τ ( k ) log + (ˆ − ˆ ) = 0 . ij ij 2 2 � n z ( k ) j = 1 ˆ ij j = 1 ν ( k ) by If the dfs are assumed to be identical, update ˆ n g ν ( k + 1 ) = argmax � ( k + 1 ) ( k + 1 ) ( k + 1 ) � ψ ( y j | ˆ � � w ( k + 1 ) , ˆ , ˆ ˆ ˆ log ξ Σ Λ , ν ) . i i i i ν j = 1 i = 1 ( NCHU ) 10 / 15 T.I. Lin National Chung Hsing University August 24, 2010

Example: The AIS data The Australian Institute of Sport (AIS) data Data : The AIS data taken by Cook and Weisberg (1994). There are 202 athletes which include 100 females and 102 males. Variables : BMI (Body mass index; kg / m 2 ) and Bfat (Body fat percentage). 35 female + male 30 25 Bfat 20 + + + + + + 15 + + + + + + + + + + + + + + + + + 10 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 5 20 25 30 35 BMI ( NCHU ) 11 / 15 T.I. Lin National Chung Hsing University August 24, 2010

Example: The AIS data A two-component MSTMIX model can be written as f ( y j | Θ ) = wf ( y j | ξ 1 , Σ 1 , Λ 1 , ν 1 ) + ( 1 − w ) f ( y j | ξ 2 , Σ 2 , Λ 2 , ν 2 ) , where � σ i , 11 � λ i , 11 � � 0 σ i , 12 and Λ i = ξ i = ( ξ i 1 , ξ i 2 ) ⊤ , Σ i = . σ i , 12 σ i , 22 0 λ i , 22 (a) (b) -1078.0 −1070 -1078.5 −1080 profile log−likelihood Profile log-likelihood -1079.0 −1090 −1100 -1079.5 −1110 -1080.0 −1120 −1130 -1080.5 50 40 30 25 30 20 20 nu2 15 10 nu1 10 5 5 10 20 30 40 50 0 0 nu Figure 2: Plot of the profile log-likelihood for ν 1 and ν 2 with a two component MSTMIX model with (a) ν 1 = ν 2 = ν (b) ν 1 � = ν 2 . ( ˆ ν 1 = 4 . 2, ˆ ν 2 = 44 . 1) ( NCHU ) 12 / 15 T.I. Lin National Chung Hsing University August 24, 2010