Gaussian Mixture Models & EM CE-717: Machine Learning Sharif - - PowerPoint PPT Presentation

Gaussian Mixture Models & EM

CE-717: Machine Learning

Sharif University of Technology

• M. Soleymani

Fall 2016

Mixture Models: definition

2

 Mixture

models: Linear supper-position

• f

mixtures

• r

components

𝑞 𝒚|𝜾 =

𝑘=1 𝐿

𝑄(𝑁

𝑘) 𝑞 𝒚 𝑁 𝑘; 𝜾𝑘

 𝑘=1 𝐿

𝑄(𝑁

𝑘) = 1  𝑄(𝑁 𝑘): the prior probability of 𝑘-th mixture  𝜾𝑘: the parameters of 𝑘-th mixture  𝑞 𝒚 𝑁 𝑘; 𝜾𝑘 : the probability of 𝒚 according to 𝑘-th mixture

 Framework for finding more complex probability distributions

 Goal: estimate 𝑞 𝒚 𝜄 E.g., Multi-modal density estimation

Gaussian Mixture Models (GMMs)

3

 Gaussian Mixture Models: 𝑞 𝒚 𝑁

𝑘; 𝜾𝑘 ~𝑂(𝝂𝑘, 𝜯𝑘)

𝑞 𝒚 =

𝑘=1 𝐿

𝜌𝑘𝒪(𝒚|𝝂𝑘, 𝜯𝑘)

 Fitting the Gaussian mixture model

 Input: data points 𝒚 𝑗 𝑗=1 𝑂  Goal: find the parameters of GMM (𝜌𝑘, 𝝂𝑘, 𝜯𝑘, 𝑘 = 1, … , 𝐿)

0 ≤ 𝜌𝑘 ≤ 1

𝑘=1 𝐿

𝜌𝑘 = 1

GMM: 1-D Example

4

2

1

   4

2 



𝜌1 = 0.6 𝜌2 = 0.3 𝜌3 = 0.1

2

1 



1

2 

 8

3 

 2 .

3 



GMM: 2-D Example

5

k = 3

𝝂1 = −2 3 Σ1 = 1 0.5 0.5 4 𝜌1 = 0.6 𝝂2 = 0 −4 Σ2 = 1 1 𝜌2 = 0.25 𝝂3 = 3 2 Σ3 = 3 1 1 1 𝜌3 = 0.15

GMM: 2-D Example

6

k = 3

𝝂1 = −2 3 Σ1 = 1 0.5 0.5 4 𝜌1 = 0.6 𝝂2 = 0 −4 Σ2 = 1 1 𝜌2 = 0.25 𝝂3 = 3 2 Σ3 = 3 1 1 1 𝜌3 = 0.15

 GMM distribution

How to Fit GMM?

7

 In order to maximize log likelihood:

ln 𝑞 𝒀 𝝆, 𝝂, 𝜯 =

𝑗=1 𝑂

ln

𝑘=1 𝑙

𝜌𝑘𝒪(𝒚|𝝂𝑘, 𝜯𝑘)

 The sum over components appears inside the log and there is no closed

form solution for maximum likelihood.

𝜖 ln 𝑞 𝒀 𝝆, 𝝂, 𝜯 𝜖𝝂𝑙 = 𝟏 𝜖 ln 𝑞 𝒀 𝝆, 𝝂, 𝜯 𝜖𝜯𝑙 = 𝟏 𝜖 ln 𝑞 𝒀 𝝆, 𝝂, 𝜯 + 𝜇 𝑘=1

𝐿

𝜌𝑘 − 1 𝜖𝜌𝑙 = 0

𝑙 = 1, … , 𝐿 𝒀 = 𝒚(1), … , 𝒚(𝑂)

ML for GMM

8

𝝂𝑙 = 1 𝑂𝑙

𝑗=1 𝑂

𝜌𝑙𝒪(𝒚(𝑗)|𝝂𝑙, 𝜯𝑙) 𝑘=1

𝐿

𝜌𝑘𝒪(𝒚(𝑗)|𝝂𝑘, 𝜯𝑘) 𝒚(𝑗) 𝜯𝑙 = 1 𝑂𝑙

𝑗=1 𝑂

𝜌𝑙𝒪(𝒚(𝑗)|𝝂𝑙, 𝜯𝑙) 𝑘=1

𝐿

𝜌𝑘𝒪(𝒚(𝑗)|𝝂𝑘, 𝜯𝑘) (𝒚(𝑗)−𝝂𝑙

new)(𝒚 𝑗 −𝝂𝑙 new)𝑈

𝜌𝑙

new = 𝑂𝑙

𝑂 𝑂𝑙 =

𝑗=1 𝑂

𝜌𝑙𝒪(𝒚(𝑗)|𝝂𝑙, 𝜯𝑙) 𝑘=1

𝐿

𝜌𝑘𝒪(𝒚(𝑗)|𝝂𝑘, 𝜯𝑘)

𝜖 log 𝑩−1 𝜖𝑩−1 = 𝑩𝑈 𝜖𝒚𝑈𝑩𝒚 𝜖𝑩 = 𝒚𝒚𝑈

EM algorithm

9

 An

iterative algorithm in which each iteration is guaranteed to improve the log-likelihood function

 General algorithm for finding ML estimation when the

data is incomplete (missing or unobserved data).

 EM find the maximum likelihood parameters in cases where

the models involve unobserved variables 𝑎 in addition to unknown parameters 𝜾 and known data observations 𝑌.

Mixture models: discrete latent variables

10

𝑞(𝒚) = 𝑄 𝑨

𝑘 = 1 𝑞 𝒚 𝑨 𝑘 = 1 = 𝑘=1 𝐿

𝜌𝑘 𝑞 𝒚 𝑨

𝑘 = 1

 𝑨: latent or hidden variable

 specifies the mixture component

 𝑄 𝑨𝑘 = 1 = 𝜌𝑘

 0 ≤ 𝜌𝑘 ≤ 1  𝑘=1

𝐿

𝜌𝑘 = 1

EM for GMM

11

 Initialize 𝝂𝑙, 𝜯𝑙, 𝜌𝑙

𝑙 = 1, … , 𝐿

 E step: 𝑗 = 1, … , 𝑂, 𝑘 = 1, … , 𝐿

𝛿𝑘

𝑗 = 𝑄 𝑨 𝑘 (𝑗) = 1|𝒚 𝑗 , 𝜾𝑝𝑚𝑒

= 𝜌𝑘

𝑝𝑚𝑒𝒪(𝒚 𝑗 |𝝂𝑘 𝑝𝑚𝑒, 𝜯𝑘 𝑝𝑚𝑒)

𝑙=1

𝐿

𝜌𝑙

𝑝𝑚𝑒𝒪(𝒚(𝑗)|𝝂𝑙 𝑝𝑚𝑒, 𝜯𝑙 𝑝𝑚𝑒)

 M Step: 𝑘 = 1, … , 𝐿

𝝂𝑘

𝑜𝑓𝑥 =

𝑗=1

𝑂

𝛿𝑘

𝑗𝒚(𝑗)

𝑗=1

𝑂

𝛿𝑘

𝑗

𝜯𝑘

𝑜𝑓𝑥 =

1 𝑗=1

𝑂

𝛿𝑘

𝑗 𝑗=1 𝑂

𝛿𝑘

𝑗(𝒚(𝑗)−𝝂𝑘 new)(𝒚 𝑗 −𝝂𝑘 new)𝑈

𝜌𝑘

new =

𝑗=1

𝑂

𝛿𝑘

𝑗

𝑂

 Repeat E and M steps until convergence

𝜾 = [𝝆, 𝝂, 𝜯] 𝑨(𝑗) ∈ {1,2, … ,𝐿} shows the mixture from which 𝑦(𝑗) is generated

EM & GMM: example

12

[Bishop]

EM & GMM: Example

13

[Bishop]

Local Minima

14

Local Minima

15

𝝂1 = −2 3 Σ1 = 1 0.5 0.5 4 𝜌1 = 0.6 𝝂2 = 0 −4 Σ2 = 1 1 𝜌2 = 0.25 𝝂3 = 3 2 Σ3 = 3 1 1 1 𝜌3 = 0.15

𝝂1 = 0.36 −4.09 Σ1 = 0.89 0.26 0.26 0.83 𝜌1 = 0.249 𝝂2 = 3.25 2.09 Σ2 = 2.23 1.08 1.09 1.41 𝜌2 = 0.146 𝝂3 = −2.11 3.36 Σ3 = 1.12 0.61 0.61 3.61 𝜌3 = 0.604 𝝂1 = 1.45 −1.81 Σ1 = 3.30 4.76 4.76 10.01 𝜌1 = 0.392 𝝂2 = −2.20 3.16 Σ2 = 1.30 1.10 1.10 2.80 𝜌2 = 0.429 𝝂3 = −1.88 3.74 Σ3 = 5.83 −0.82 −0.82 5.83 𝜌3 = 0.178

𝐷1 𝐷2 𝐷3 𝐷1 𝐷2 𝐷3

EM+GMM vs. k-means

16

 k-means:

 It is not probabilistic  Has fewer parameters (and faster)  Limited by the underlying assumption of spherical clusters

 can be extended to use covariance – get “hard EM” (ellipsoidal k-

means).  Both EM and k-means depend on initialization

 getting stuck in local optima

 EM+GMM has more local minima  Useful trick: first run k-means and then use its result to initialize EM.

EM algorithm: general

General algorithm for finding ML estimation when the data is incomplete (missing or unobserved data).

Incomplete log likelihood

18

 Complete log likelihood

 Maximizing likelihood (i.e., log 𝑄(𝑌, 𝑍|𝜾)) for labeled data is

straightforward

 Incomplete log likelihood

 With 𝑎 unobserved, our objective becomes the log of a

marginal probability log 𝑄(𝑌|𝜾) = log 𝑎 𝑄(𝑌, 𝑎|𝜾)

 This objective will not decouple and we use EM algorithm to solve it

EM Algorithm

19

 Assumptions: 𝑌 (observed or known variables), 𝑎 (unobserved or latent

variables), 𝑌 come from a specific model with unknown parameters 𝜾

 If 𝑎 is relevant to 𝑌 (in any way), we can hope to extract information about it

from 𝑌 assuming a specific parametric model on the data.  Steps:

 Initialization: Initialize the unknown parameters 𝜾  Iterate the following steps, until convergence:

 Expectation step: Find the probability of unobserved variables given the current

parameters estimates and the observed data.

 Maximization step: from the observed data and the probability of the

unobserved data find the most likely parameters (a better estimate for the parameters).

EM algorithm intuition

20

 When learning with hidden variables, we are trying to solve

two problems at once:

 hypothesizing values for the unobserved variables in each data sample  learning the parameters

 Each of these tasks is fairly easy when we have the solution to

the other.

 Given complete data, we have the statistics, and we can estimate

parameters using the MLE formulas.

 Conversely, computing probability of missing data given the parameters is

a probabilistic inference problem

EM algorithm

21

EM theoretical analysis

22

 What is the underlying theory for the use of the

expected complete log likelihood in the M-step? 𝐹𝑄 𝑎 𝑌, 𝜾𝑝𝑚𝑒 log 𝑄 𝑌, 𝑎 𝜾

 Now, we

show that maximizing this function also maximizes the likelihood

EM theoretical foundation: Objective function

23

𝑎

Jensen’s inequality

24

EM theoretical foundation: Algorithm in general form

25

EM theoretical foundation: E-step

26

𝑅𝑢 = 𝑄(𝑎|𝑌, 𝜾𝑢) ⟹ 𝑅𝑢 = argmax

𝑅

𝐺 𝜾𝑢, 𝑅 Proof: 𝐺 𝜾𝑢, 𝑄(𝑎|𝑌, 𝜾𝑢) =

𝑎

𝑄(𝑎|𝑌, 𝜾𝑢) log 𝑄(𝑌, 𝑎|𝜾𝑢) 𝑄(𝑎|𝑌, 𝜾𝑢) =

𝑎

𝑄(𝑎|𝑌, 𝜾𝑢) log 𝑄(𝑌|𝜾𝑢) = log 𝑄(𝑌|𝜾𝑢) = ℓ 𝜾𝑢; 𝑌

 𝐺 𝜾, 𝑅

is a lower bound on ℓ 𝜾; 𝑌 . Thus, 𝐺 𝜾𝑢, 𝑅 has been maximized by setting 𝑅 to 𝑄 𝑎 𝑌, 𝜾𝑢 : 𝐺 𝜾𝑢, 𝑄(𝑎|𝑌, 𝜾𝑢) = ℓ 𝜾𝑢; 𝑌 ⇒ 𝑄 𝑎 𝑌, 𝜾𝑢 = argmax

𝑅

𝐺 𝜾𝑢, 𝑅

EM algorithm: illustration

27

ℓ 𝜾; 𝑌 𝐺 𝜾, 𝑅𝑢 𝜾𝑢 𝜾𝑢+1

EM theoretical foundation: M-step

28

M-step can be equivalently viewed as maximizing the expected complete log-likelihood: 𝜾𝑢+1 = argmax

𝜾

𝐺 𝜾, 𝑅𝑢 = argmax

𝜾

𝐹𝑅𝑢 log 𝑄(𝑌, 𝑎|𝜾) Proof:

𝐺 𝜾, 𝑅𝑢 =

𝑎

𝑅𝑢(𝑎) log 𝑄(𝑌, 𝑎|𝜾) 𝑅𝑢(𝑎) =

𝑎

𝑅𝑢(𝑎) log 𝑄(𝑌, 𝑎|𝜾) −

𝑎

𝑅𝑢(𝑎) log 𝑅𝑢(𝑎) ⇒ 𝐺 𝜾, 𝑅𝑢 = 𝐹𝑅𝑢 log 𝑄(𝑌, 𝑎|𝜾) + 𝐼(𝑅𝑢 𝑎 )

Independent of 𝜾

EM iteration increases ℓ 𝜾; 𝑌

29

ℓ 𝜾𝑢; 𝑌 = 𝐹𝑅𝑢 log 𝑄 𝑌, 𝑎 𝜾𝑢 + 𝐼(𝑅𝑢 𝑎 ) ℓ 𝜾𝑢+1; 𝑌 ≥ 𝐹𝑅𝑢 log 𝑄 𝑌, 𝑎 𝜾𝑢+1 + 𝐼(𝑅𝑢 𝑎 ) ℓ 𝜾𝑢+1; 𝑌 − ℓ 𝜾𝑢; 𝑌 ≥ 𝐹𝑅𝑢 log 𝑄 𝑌, 𝑎 𝜾𝑢+1 − 𝐹𝑅𝑢 log 𝑄 𝑌, 𝑎 𝜾𝑢 Moreover, we have: 𝜾𝑢+1 = argmax

𝜾

𝐹𝑅𝑢 log 𝑄 𝑌, 𝑎 𝜾 ⇒ 𝐹𝑅𝑢 log 𝑄 𝑌, 𝑎 𝜾𝑢+1 ≥ 𝐹𝑅𝑢 log 𝑄 𝑌, 𝑎 𝜾𝑢 ⇒ ℓ 𝜾𝑢+1; 𝑌 − ℓ 𝜾𝑢; 𝑌 ≥ 0 EM is guaranteed to find a local maxima of the log likelihood

30

EM for GMM M step: details

31

𝑞 𝑌, 𝑎 𝜾 =

𝑗=1 𝑂

𝑞(𝒚 𝑗 , 𝒜 𝑗 |𝜾) =

𝑗=1 𝑂

𝑞(𝒚 𝑗 |𝒜 𝑗 , 𝜾)𝑞(𝒜 𝑗 |𝝆) =

𝑗=1 𝑂 𝑘=1 𝐿

𝒪 𝒚 𝑗 𝝂𝑘 , 𝜯𝑘

𝑨𝑘

(𝑗)

𝜌𝑘

𝑨𝑘

(𝑗)

log 𝑞 𝑌, 𝑎 𝜾 =

𝑗=1 𝑂 𝑘=1 𝐿

𝑨

𝑘 (𝑗) log 𝒪 𝒚 𝑗 𝝂𝑘 , 𝜯𝑘

+ log 𝜌𝑘 𝐹𝑎~𝑄 𝑎 𝑌, 𝜾old log 𝑞 𝑌, 𝑎 𝜾 = =

𝑗=1 𝑂 𝑘=1 𝐿

𝐹𝑄 𝑨𝑘

𝑗 |𝒚 𝑗 ,𝜾old

𝑨

𝑘 𝑗

log 𝒪 𝒚 𝑗 𝝂𝑘 , 𝜯𝑘 + log 𝜌𝑘

𝜾 = [𝝆, 𝝂, 𝜯] 𝜾𝑝𝑚𝑒 = [𝝆𝑝𝑚𝑒, 𝝂𝑝𝑚𝑒, 𝜯𝐩𝐦𝐞] 𝛿𝑘

𝑗

EM for GMM M step: details

32

𝜖𝑅 𝜾; 𝜾𝒑𝒎𝒆 𝜖𝝂𝑘 = 0 ⇒ 𝝂𝑘 = 𝑗=1

𝑂

𝛿𝑘

𝑗𝒚(𝑗)

𝑗=1

𝑂

𝛿𝑘

𝑗

𝜖𝑅 𝜾; 𝜾𝒑𝒎𝒆 𝜖𝜯𝑘 = 0 ⇒ 𝜯𝑘 = 1 𝑗=1

𝑂

𝛿𝑘

𝑗 𝑗=1 𝑂

𝛿𝑘

𝑗(𝒚(𝑗)−𝝂𝑘 )(𝒚 𝑗 −𝝂𝑘 )𝑈

𝜖 𝑅 𝜾; 𝜾𝒑𝒎𝒆 + 𝜇 𝑚=1

𝑙

𝜌𝑚 − 1 𝜖𝜌𝑘 = 0 ⇒ 𝜌𝑘 = 𝑗=1

𝑂

𝛿𝑘

𝑗

𝑂

Lagrange multiplier due to the constraint 𝑘=1

𝑙

𝜌𝑘 = 1

EM algorithm: general

33

 EM: general procedure for learning from partly observed

data

 Define: Q(𝜾; 𝜾old) = 𝐹𝑎~𝑄(𝑎|𝑌,𝜾old) log 𝑞(𝑌, 𝑎|𝜾) 

= 𝑎 𝑄(𝑎|𝑌, 𝜾old) × log 𝑞(𝑌, 𝑎|𝜾)

Choose an initial setting 𝜾old = 𝜾0 Iterate until convergence: E Step: Use 𝑌 and current 𝜾old to calculate 𝑄(𝑎|𝑌, 𝜾old) M Step: 𝜾new = argmax

𝜾

Q(𝜾; 𝜾old) 𝜾old ← 𝜾new

expectation of the log-likelihood evaluated using the current estimate for the parameters 𝜾old

EM advantages and disadvantages

34

 Some good things about EM:

 no learning rate (step-size) parameter  automatically enforces parameter constraints  very fast for low dimensions  each iteration guaranteed to improve likelihood

 Some bad things about EM:

 can be slower than some other iterative gradient-based

methods

Semi-supervised learning

35

 Supervised Learning models require labeled data

 Supervised learning usually requires plenty of labeled data

 It is usually expensive to have a large set of labeled data  Unlabeled data is often abundant with no or low cost

 Learning from both labeled and unlabeled data

 Labeled training data: ℒ =

𝒚 𝑜 , 𝑧 𝑜

𝑚=1 𝑀

 Unlabeled data available during training: 𝒱 = 𝒚 𝑜

𝑜=𝑀+1 𝑀+𝑉

Semi-supervised learning: example

36

Zhu, Semi-Supervised Learning Tutorial, ICML, 2007.

37

Zhu, Semi-Supervised Learning Tutorial, ICML, 2007.

Semi-supervised generative model

38

 Start from MLE 𝜾 = 𝝆, 𝝂, 𝜯 on ℒ =

𝒚 𝑜 , 𝑧 𝑜

𝑚=1 𝑀

 Repeat:

 E-step: compute 𝑞(𝑧 𝑜 |𝒚(𝑜), 𝜾) for 𝑜 = 𝑀 + 1 to 𝑜 = 𝑀 + 𝑉  M-step: compute the parameters 𝜾 = 𝝆, 𝝂, 𝜯

considering both labeled data and unlabeled data using the distribution found on their labels in the E-step

Resource

39

 C. Bishop, “Pattern Recognition and Machine Learning”,

Chapter 9.