Mixture Models Simulation-based Estimation Michel Bierlaire - PowerPoint PPT Presentation

Mixture Models — Simulation-based Estimation Michel Bierlaire Transport and Mobility Laboratory School of Architecture, Civil and Environmental Engineering Ecole Polytechnique F´ ed´ erale de Lausanne M. Bierlaire (TRANSP-OR ENAC EPFL) Mixture Models — Simulation-based Estimation 1 / 78

Outline Outline Mixtures 1 Relaxing the independence assumption 2 Relaxing the identical distribution assumption 3 Taste heterogeneity 4 Latent classes 5 Summary 6 M. Bierlaire (TRANSP-OR ENAC EPFL) Mixture Models — Simulation-based Estimation 2 / 78

Mixtures Definition Mixtures Mixture probability distribution function Convex combination of other probability distribution functions. Property Let f ( ε, θ ) be a parametrized family of distribution functions Let w ( θ ) be a non negative function such that � w ( θ ) d θ = 1 θ Then � g ( ε ) = w ( θ ) f ( ε, θ ) d θ θ is also a distribution function. M. Bierlaire (TRANSP-OR ENAC EPFL) Mixture Models — Simulation-based Estimation 3 / 78

Mixtures Definition Mixtures We say that g is a w -mixture of f If f is a logit model, g is a continuous w -mixture of logit If f is a MEV model, g is a continuous w -mixture of MEV M. Bierlaire (TRANSP-OR ENAC EPFL) Mixture Models — Simulation-based Estimation 4 / 78

Mixtures Definition Mixtures Discrete mixtures If w i , i = 1 , . . . , n are non negative weights such that n � w i = 1 i =1 then n � g ( ε ) = w i f ( ε, θ i ) i =1 is also a distribution function where θ i , i = 1 , . . . , n are parameters. We say that g is a discrete w -mixture of f . M. Bierlaire (TRANSP-OR ENAC EPFL) Mixture Models — Simulation-based Estimation 5 / 78

Mixtures Definition Example: discrete mixture of normal distributions 2.5 N(5,0.16) N(8,1) 0.6 N(5,0.16) + 0.4 N(8,1) 2 1.5 1 0.5 0 4 5 6 7 8 9 10 11 M. Bierlaire (TRANSP-OR ENAC EPFL) Mixture Models — Simulation-based Estimation 6 / 78

Mixtures Definition Example: discrete mixture of binary logit models 1 P(1|s=1,x) P(1|s=2,x) 0. 4 P(1|s=1,x) + 0.6 P(1|s=2,x) 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 -4 -2 0 2 4 M. Bierlaire (TRANSP-OR ENAC EPFL) Mixture Models — Simulation-based Estimation 7 / 78

Mixtures Definition Mixtures General motivation Generate flexible distributional forms For discrete choice correlation across alternatives alternative specific variances taste heterogeneity . . . M. Bierlaire (TRANSP-OR ENAC EPFL) Mixture Models — Simulation-based Estimation 8 / 78

Mixtures Mixtures of logit Continuous Mixtures of logit Combining probit and logit Error components U in = V in + ξ in + ν in i.i.d EV (logit): tractability Normal distribution (probit): flexibility M. Bierlaire (TRANSP-OR ENAC EPFL) Mixture Models — Simulation-based Estimation 9 / 78

Mixtures Mixtures of logit Logit Specification of the utility functions U auto = β X auto + ν auto = β X bus + ν bus U bus U subway = β X subway + ν subway Distributional assumption ν i.i.d. extreme value Choice model e β X auto Pr(auto | X , C ) = e β X auto + e β X bus + e β X subway M. Bierlaire (TRANSP-OR ENAC EPFL) Mixture Models — Simulation-based Estimation 10 / 78

Mixtures Mixtures of logit Normal mixture of logit Specification of the utility functions = β X auto + ξ auto + ν auto U auto U bus = β X bus + ξ bus + ν bus = β X subway + ξ subway + ν subway U subway Distributional assumptions ν i.i.d. extreme value ξ ∼ N (0 , Σ) Choice model e β X auto + ξ auto Pr(auto | X , ξ ) = e β X auto + ξ auto + e β X bus + ξ bus + e β X subway + ξ subway � P (auto | X ) = Pr(auto | X , ξ ) f ( ξ ) d ξ ξ M. Bierlaire (TRANSP-OR ENAC EPFL) Mixture Models — Simulation-based Estimation 11 / 78

Mixtures Simulation Calculation Choice model � P (auto | X ) = Pr(auto | X , ξ ) f ( ξ ) d ξ ξ Calculation Integral has no closed form. If one dimension is involved, numerical integration can be used. With more dimensions, Monte Carlo simulation must be used. M. Bierlaire (TRANSP-OR ENAC EPFL) Mixture Models — Simulation-based Estimation 12 / 78

Mixtures Simulation Simulation In order to approximate � P (auto | X ) = Pr(auto | X , ξ ) f ( ξ ) d ξ ξ Draw from f ( ξ ) to obtain r 1 , . . . , r R Compute P (auto | X ) ≈ ˜ P (auto | X ) = 1 � R k =1 P (auto | X , r k ) = R R e β X auto + r 1 k 1 � e β X auto + r 1 k + e β X bus + r 2 k + e β X subway + r 3 k R k =1 M. Bierlaire (TRANSP-OR ENAC EPFL) Mixture Models — Simulation-based Estimation 13 / 78

Mixtures Simulation Simulation Can approximate as close as needed R 1 � P (auto | X ) = lim P (auto | X , r k ) . R R →∞ k =1 In practice Efficient methods to draw from the distribution. R must be large enough. M. Bierlaire (TRANSP-OR ENAC EPFL) Mixture Models — Simulation-based Estimation 14 / 78

Relaxing the independence assumption Outline Mixtures 1 Relaxing the independence assumption 2 Nesting Cross-nesting Relaxing the identical distribution assumption 3 Taste heterogeneity 4 Latent classes 5 Summary 6 M. Bierlaire (TRANSP-OR ENAC EPFL) Mixture Models — Simulation-based Estimation 15 / 78

Relaxing the independence assumption Nesting Capturing correlations: nesting Specification of the utility functions = β X auto + ν auto U auto U bus = β X bus + σ transit η transit + ν bus = β X subway + σ transit η transit + ν subway U subway Distributional assumptions ν i.i.d. extreme value, η transit ∼ N (0 , 1), σ 2 transit =cov(bus,subway) Choice model e β X auto Pr(auto | X , η transit ) = e β X auto + e β X bus + σ transit η transit + e β X subway + σ transit η transit � P (auto | X ) = Pr(auto | X , η ) f ( η ) d η M. Bierlaire (TRANSP-OR ENAC EPFL) Mixture Models — Simulation-based Estimation 16 / 78 η

Relaxing the independence assumption Nesting Nesting structure Example: residential telephone Ct. BM Ct. SM Ct. LF Ct. EF β C σ M σ F BM 1 0 0 0 ln(cost(BM)) η M 0 SM 0 1 0 0 ln(cost(SM)) η M 0 LF 0 0 1 0 ln(cost(LF)) 0 η F EF 0 0 0 1 ln(cost(EF)) 0 η F MF 0 0 0 0 ln(cost(MF)) 0 η F M. Bierlaire (TRANSP-OR ENAC EPFL) Mixture Models — Simulation-based Estimation 17 / 78

Relaxing the independence assumption Nesting Nesting structure Identification issues If there are two nests, only one σ is identified If there are more than two nests, all σ ’s are identified Walker (2001) M. Bierlaire (TRANSP-OR ENAC EPFL) Mixture Models — Simulation-based Estimation 18 / 78

Relaxing the independence assumption Nesting Results with 5000 draws NL NML NML NML NML σ F = 0 σ M = 0 σ F = σ M -473.219 -472.768 -473.146 -472.779 -472.846 L Estim. Scaled Estim. Scaled Estim. Scaled Estim. Scaled Estim. Scaled Ct .BM -1.78 1.00 -3.81 1.00 -3.79 1.00 -3.81 1.00 -3.81 1.00 Ct. EF -0.558 0.313 -1.20 0.314 -1.19 0.313 -1.20 0.314 -1.20 0.314 Ct. LF -0.512 0.287 -1.10 0.287 -1.09 0.287 -1.09 0.287 -1.09 0.287 Ct. SM -1.41 0.788 -3.02 0.791 -3.00 0.790 -3.01 0.791 -3.02 0.791 β C -1.49 0.835 -3.26 0.855 -3.24 0.855 -3.26 0.855 -3.26 0.854 2.29 µ FLAT 2.06 µ MEAS σ F 3.02 0.00 3.06 2.17 σ M 0.530 3.02 0.00 2.17 σ 2 F + σ 2 9.40 9.15 9.37 9.43 M M. Bierlaire (TRANSP-OR ENAC EPFL) Mixture Models — Simulation-based Estimation 19 / 78

Relaxing the independence assumption Nesting Comments The scale of the parameters is different between NL and the mixture model Normalization can be performed in several ways σ F = 0 σ M = 0 σ F = σ M Final log likelihood should be the same But... estimation relies on simulation Only an approximation of the log likelihood is available Final log likelihood with 50000 draws: Unnormalized: -472.872 σ M = σ F : -472.875 σ F = 0: -472.884 σ M = 0: -472.901 M. Bierlaire (TRANSP-OR ENAC EPFL) Mixture Models — Simulation-based Estimation 20 / 78

Relaxing the independence assumption Cross-nesting Cross nesting Motorized Private Bus Train Car Ped. Bike M. Bierlaire (TRANSP-OR ENAC EPFL) Mixture Models — Simulation-based Estimation 21 / 78

Relaxing the independence assumption Cross-nesting Cross nesting Specification U bus = V bus + ξ 1 + ν bus = + ξ 1 + ν train U train V train U car = V car + ξ 1 + ξ 2 + ν car = + ξ 2 + ν ped U ped V ped U bike = V bike + ξ 2 + ν bike Choice model � � P (car) = P (car | ξ 1 , ξ 2 ) f ( ξ 1 ) f ( ξ 2 ) d ξ 2 d ξ 1 ξ 1 ξ 2 M. Bierlaire (TRANSP-OR ENAC EPFL) Mixture Models — Simulation-based Estimation 22 / 78

Relaxing the independence assumption Cross-nesting Identification issue Not all parameters can be identified For logit, one ASC has to be constrained to zero Identification of NML is important and tricky See Walker, Ben-Akiva & Bolduc (2007) for a detailed analysis M. Bierlaire (TRANSP-OR ENAC EPFL) Mixture Models — Simulation-based Estimation 23 / 78

Relaxing the identical distribution assumption Outline Mixtures 1 Relaxing the independence assumption 2 Relaxing the identical distribution assumption 3 Normalization Taste heterogeneity 4 Latent classes 5 Summary 6 M. Bierlaire (TRANSP-OR ENAC EPFL) Mixture Models — Simulation-based Estimation 24 / 78