Nonparametric spectral-based estimation of latent structures Stéphane Bonhomme (Chicago), Koen Jochmans (Sciences Po) and J.-M. Robin (Sciences Po and UCL) May 27, 2014 1 / 29 Bonhomme, Jochmans, Robin Nonparametric spectral-based estimation of latent structures
Paper question Economist like unobserved heterogeneity and dynamic factor models. Usually discrete mixtures of parametric distributions (derived from theory) For identification and also estimation, it is useful to consider discrete mixtures of nonparametric models. This paper proposes a simple estimation procedure for discrete mixtures and hidden Markov models of nonparametric distribution components. 2 / 29 Bonhomme, Jochmans, Robin Nonparametric spectral-based estimation of latent structures
Identification The question of identification in latent structures is the topic of a very recent and active literature. Nonparametric identification from univariate/cross-sectional data typically fails. (Some exceptions for location models) Multivariate data (panel data) can present a powerful identification source. Finite mixtures/latent-class models: Hall and Zhou (2003); 1 Allman et al. (2009) (Dynamic) discrete-choice models: Magnac and Thesmar (2002); 2 Kasahara and Shimotsu (2009) Hidden Markov/regime-switching models: Allman et al. (2009); 3 Gassiat et al. (2013) Models for corrupted and misclassified data: Schennach (2004); 4 Hu and Schennach (2008) 3 / 29 Bonhomme, Jochmans, Robin Nonparametric spectral-based estimation of latent structures
Contribution We propose a new constructive identification argument... that delivers a least square-type estimator for mixture weights... allowing for asymptotic distributional theory. 4 / 29 Bonhomme, Jochmans, Robin Nonparametric spectral-based estimation of latent structures
Discrete mixtures of discrete distributions Let ( y 1 ,..., y q ) be q discrete variables with supp ( y i ) = { 1 ,..., κ i } . There exists a latent variable x ∈ { 1 ,..., r } with π j ≡ Pr { x = j } . Let p ij ∈ [ 0 , 1 ] κ i denote the vector of conditional probability masses of y i given x = j : p ij ( k ) ≡ Pr { y i = k | x = j } , k = 1 ,..., κ i 5 / 29 Bonhomme, Jochmans, Robin Nonparametric spectral-based estimation of latent structures
Unconditional distribution for DMs The unconditional joint PDF of ( y 1 ,..., y q ) is r ∑ P ( y 1 ,..., y q ) = π j p 1 j ( y 1 ) p 2 j ( y 2 ) ... p qj ( y q ) j = 1 The set of values P ( y 1 ,..., y q ) for all ( y 1 ,..., y q ) defines a q -dimensional array r ∑ P = π j p 1 j � p 2 j � ··· � p qj j = 1 � is the Kronecker product 6 / 29 Bonhomme, Jochmans, Robin Nonparametric spectral-based estimation of latent structures
Hidden Markov models There are q discrete latent variables ( x 1 ,..., x q ) for q measurements ( y 1 ,..., y q ) . Stationarity: Pr { x i = j } = π j , i = 1 ,..., q Pr { x i + 1 | x i } = K ( x i , x i + 1 ) , i = 1 ,..., q − 1 Pr { y i = k | x i = j } = p j ( k ) , k = 1 ,..., κ Conditional independence: measurements y 1 ,..., y q are independent conditional on ( x 1 ,..., x q ) . 7 / 29 Bonhomme, Jochmans, Robin Nonparametric spectral-based estimation of latent structures
Unconditional distribution for HHMs (1) The unconditional joint PDF of ( y 1 ,..., y 3 ) is P ( y 1 , y 2 , y 3 ) � � �� r r r ∑ ∑ ∑ = π j 1 p j 1 ( y 1 ) K ( j 1 , j 2 ) p j 2 ( y 2 ) K ( j 2 , j 3 ) p j 3 ( y 3 ) j 1 = 1 j 2 = 1 j 3 = 1 �� � � �� r r r ∑ ∑ ∑ = p j 1 ( y 1 ) π j 1 K ( j 1 , j 2 ) p j 2 ( y 2 ) K ( j 2 , j 3 ) p j 3 ( y 3 ) j 2 = 1 j 1 = 1 j 3 = 1 8 / 29 Bonhomme, Jochmans, Robin Nonparametric spectral-based estimation of latent structures
Unconditional distribution for HHMs (2) Let P = [ p 1 ,..., p r ] ∈ R κ × r and Π = diag ( π 1 ,..., π r ) . Hence the 3-dimensional array r � PK ⊤ � ∑ P = ( P Π K ) j � p j � j j = 1 where M j denotes the j th column of matrix M If q > 3 one can select all consecutive triples or regroup observations into 3 consecutive clusters: ( y 1 ,..., y k − 1 ) , y k , ( y k + 1 ,..., y q ) . 9 / 29 Bonhomme, Jochmans, Robin Nonparametric spectral-based estimation of latent structures
Identification of such latent array structures Kruskal (Psychometrica 1976, Linear Algebra Appl. 1977) Consider a κ 1 × κ 2 × κ 3 array P = ∑ r j = 1 p 1 j � p 2 j � p 3 j Let P i = [ p i 1 ,..., p ir ] ∈ R κ i × r , i = 1 , 2 , 3 Let r i = max { k : all collections of k columns of P i are independent} (the Kruskal-rank of P i ). Note that if P ∈ R κ × r has rank r it also has Kruskal-rank r . If r 1 + r 2 + r 3 ≥ 2 r + 2 then P uniquely determines the matrices P i up to simultaneous column-permutation and common column-scaling. 10 / 29 Bonhomme, Jochmans, Robin Nonparametric spectral-based estimation of latent structures
Application to statistics Allman, Matias and Rhodes (AoS, 2009) Allman et al. use Kruskal’s result to give conditions for the identification of discrete mixtures of discrete and continuous nonparametric distributions, hidden Markov models and some stochastic graphs. 11 / 29 Bonhomme, Jochmans, Robin Nonparametric spectral-based estimation of latent structures
Discrete mixtures Allman, Matias and Rhodes (AoS, 2009) Kruskal’s theorem applies with r ∑ P = π j p 1 j � p 2 j � p 3 j j = 1 P 1 = [ π 1 p 11 ,..., π r p 1 r ] , P i = [ p i 1 ,..., p ir ] , i > 1 Since sum ( P 1 , 1 ) = [ π 1 ,..., π r ] and (Corollary 2) sum ( P i , 1 ) = [ 1 ,..., 1 ] , i > 1 , then, if r 1 + r 2 + r 3 ≥ 2 r + 2 , group-probabilities π j and conditional probabilities p ij are identified up to labeling. (Theorem 8) Holds for continuous mixture components if the component densities are linearly independent ( r 1 = r 2 = r 3 = r ). 12 / 29 Bonhomme, Jochmans, Robin Nonparametric spectral-based estimation of latent structures
HMMs Allman, Matias and Rhodes (AoS, 2009), Theorem 6 The parameters of an HMM with r hidden states and κ observable states are generically identifiable from the marginal distribution of 2 k + 1 consecutive variables provided k satisfies � k + κ − 1 � ≥ r κ − 1 � k + κ − 1 � Note that = κ for k = 1 (3 measurements) and κ − 1 � k + κ − 1 � = k + 1 for κ = 2 (binary outcomes). κ − 1 13 / 29 Bonhomme, Jochmans, Robin Nonparametric spectral-based estimation of latent structures
Application to HMMs Gassiat, Cleynen, Robin (arXiv, 2013), Theorem 2.1 They use Allman et al.’s result to prove the following result. Assume that r is known, P = [ p 1 ,..., p r ] is full column rank, and K has full rank. Then K and P are identifiable from from the distribution of 3 consecutive observations ( y 1 , y 2 , y 3 ) up to label swapping of the hidden states. Estimation by penalized ML or EM algorithm. 14 / 29 Bonhomme, Jochmans, Robin Nonparametric spectral-based estimation of latent structures
Constructive identification procedures There exists few constructive identification procedures. De Lathauwer (SIAM, 2006) applies to the case where one outcome (say y 1 ) is such that P 1 is full column rank. However it provides identification only up to relabeling AND scaling. Group probabilities π j are thus not identified. We propose one such constructive identification that works both for DMs and HMMs, inspired from ICA or blind deconvolution. 15 / 29 Bonhomme, Jochmans, Robin Nonparametric spectral-based estimation of latent structures
DMs P = ∑ r j = 1 π j p 1 j � p 2 j � p 3 j Let Π = diag ( π 1 ,..., π r ) , and P i = [ p i 1 ,..., p ir ] ∈ R κ i × r , i = 1 , 2 , 3 . Assume rank ( P i ) = r and π j > 0 . 16 / 29 Bonhomme, Jochmans, Robin Nonparametric spectral-based estimation of latent structures
DMs P 1 Π P ⊤ 2 = ∑ r j = 1 π j p 1 j p ⊤ 2 j = ∑ r j = 1 π j p 1 j � p 2 j is the matrix containing probabilities P ( y 1 , y 2 ) . Observable. SVD on P 1 Π P ⊤ 2 , which has rank r , allows to construct U and V such that κ 2 × r = I r ⇒ ( VP 2 ) ⊤ = ( UP 1 Π) − 1 ≡ Q − 1 r × κ 1 P 1 Π P ⊤ 2 V ⊤ U r × r P (: , : , k ) = ∑ r j = 1 π j p 1 j � p 2 j � p 3 j ( k ) = P 1 Π D 3 k P ⊤ 2 , with D 3 k = diag [ p 31 ( k ) ,..., p 3 r ( k )] , is the matrix containing probabilities P ( y 1 , y 2 , k ) (for any y 1 , y 2 and y 3 = k ). Also observable. W k = U P (: , : , k ) V ⊤ = QD 3 k Q − 1 (whitening) P 3 identified by the eigenvalues of matrices W 1 ,..., W κ 3 Repeat for P 1 and P 2 . π = [ π 1 ; ... ; π r ] identified from P ( y i ) = ∑ r j = 1 π j p ij ( y i ) = P π 17 / 29 Bonhomme, Jochmans, Robin Nonparametric spectral-based estimation of latent structures
Recommend
More recommend
