Lecture 7: Multiclass Classification Princeton University COS 495 - - PowerPoint PPT Presentation

Machine Learning Basics Lecture 7: Multiclass Classification

Princeton University COS 495 Instructor: Yingyu Liang

Example: image classification

indoor

• utdoor

Indoor

Example: image classification (multiclass)

ImageNet figure borrowed from vision.standford.edu

Multiclass classification

• Given training data 𝑦𝑗, 𝑧𝑗 : 1 ≤ 𝑗 ≤ 𝑜 i.i.d. from distribution 𝐸
• 𝑦𝑗 ∈ 𝑆𝑒, 𝑧𝑗∈ {1,2, … , 𝐿}
• Find 𝑔 𝑦 : 𝑆𝑒 → {1,2, … , 𝐿} that outputs correct labels
• What kind of 𝑔?

Approaches for multiclass classification

Approach 1: reduce to regression

• Given training data 𝑦𝑗, 𝑧𝑗 : 1 ≤ 𝑗 ≤ 𝑜 i.i.d. from distribution 𝐸
• Find 𝑔

𝑥 𝑦 = 𝑥𝑈𝑦 that minimizes ෠

𝑀 𝑔

𝑥 = 1 𝑜 σ𝑗=1 𝑜

𝑥𝑈𝑦𝑗 − 𝑧𝑗 2

• Bad idea even for binary classification

Reduce to linear regression; ignore the fact 𝑧 ∈ {1,2. . . , 𝐿}

Approach 1: reduce to regression

Figure from Pattern Recognition and Machine Learning, Bishop

Bad idea even for binary classification

Approach 2: one-versus-the-rest

• Find 𝐿 − 1 classifiers 𝑔

1, 𝑔 2, … , 𝑔 𝐿−1

• 𝑔

1 classifies 1 𝑤𝑡 {2,3, … , 𝐿}

• 𝑔

2 classifies 2 𝑤𝑡 {1,3, … , 𝐿}

• …
• 𝑔

𝐿−1 classifies 𝐿 − 1 𝑤𝑡 {1,2, … , 𝐿 − 2}

• Points not classified to classes {1,2, … , 𝐿 − 1} are put to class 𝐿
• Problem of ambiguous region: some points may be classified to more

than one classes

Approach 2: one-versus-the-rest

Figure from Pattern Recognition and Machine Learning, Bishop

Approach 3: one-versus-one

• Find 𝐿 − 1 𝐿/2 classifiers 𝑔

(1,2), 𝑔 (1,3), … , 𝑔 (𝐿−1,𝐿)

• 𝑔

(1,2) classifies 1 𝑤𝑡 2

• 𝑔

(1,3) classifies 1 𝑤𝑡 3

• …
• 𝑔

(𝐿−1,𝐿) classifies 𝐿 − 1 𝑤𝑡 𝐿

• Computationally expensive: think of 𝐿 = 1000
• Problem of ambiguous region

Approach 3: one-versus-one

Figure from Pattern Recognition and Machine Learning, Bishop

Approach 4: discriminant functions

• Find 𝐿 scoring functions 𝑡1, 𝑡2, … , 𝑡𝐿
• Classify 𝑦 to class 𝑧 = argmax𝑗 𝑡𝑗(𝑦)
• Computationally cheap
• No ambiguous regions

Linear discriminant functions

• Find 𝐿 discriminant functions 𝑡1, 𝑡2, … , 𝑡𝐿
• Classify 𝑦 to class 𝑧 = argmax𝑗 𝑡𝑗(𝑦)
• Linear discriminant: 𝑡𝑗(𝑦) = 𝑥𝑗 𝑈𝑦, with 𝑥𝑗 ∈ 𝑆𝑒

Linear discriminant functions

• Linear discriminant: 𝑡𝑗(𝑦) = 𝑥𝑗 𝑈𝑦, with 𝑥𝑗 ∈ 𝑆𝑒
• Lead to convex region for each class: by 𝑧 = argmax𝑗 𝑥𝑗 𝑈𝑦

Figure from Pattern Recognition and Machine Learning, Bishop

Conditional distribution as discriminant

• Find 𝐿 discriminant functions 𝑡1, 𝑡2, … , 𝑡𝐿
• Classify 𝑦 to class 𝑧 = argmax𝑗 𝑡𝑗(𝑦)
• Conditional distributions: 𝑡𝑗(𝑦) = 𝑞(𝑧 = 𝑗|𝑦)
• Parametrize by 𝑥𝑗: 𝑡𝑗(𝑦) = 𝑞𝑥𝑗(𝑧 = 𝑗|𝑦)

Multiclass logistic regression

Review: binary logistic regression

• Sigmoid

𝜏 𝑥𝑈𝑦 + 𝑐 = 1 1 + exp(−(𝑥𝑈𝑦 + 𝑐))

• Interpret as conditional probability

𝑞𝑥 𝑧 = 1 𝑦 = 𝜏 𝑥𝑈𝑦 + 𝑐 𝑞𝑥 𝑧 = 0 𝑦 = 1 − 𝑞𝑥 𝑧 = 1 𝑦 = 1 − 𝜏 𝑥𝑈𝑦 + 𝑐

• How to extend to multiclass?

Review: binary logistic regression

• Suppose we model the class-conditional densities 𝑞 𝑦 𝑧 = 𝑗 and

class probabilities 𝑞 𝑧 = 𝑗

• Conditional probability by Bayesian rule:

𝑞 𝑧 = 1|𝑦 = 𝑞 𝑦|𝑧 = 1 𝑞(𝑧 = 1) 𝑞 𝑦|𝑧 = 1 𝑞 𝑧 = 1 + 𝑞 𝑦|𝑧 = 2 𝑞(𝑧 = 2) = 1 1 + exp(−𝑏) = 𝜏(𝑏)

where we define

𝑏 ≔ ln 𝑞 𝑦|𝑧 = 1 𝑞(𝑧 = 1) 𝑞 𝑦|𝑧 = 2 𝑞(𝑧 = 2) = ln 𝑞 𝑧 = 1|𝑦 𝑞 𝑧 = 2|𝑦

Review: binary logistic regression

• Suppose we model the class-conditional densities 𝑞 𝑦 𝑧 = 𝑗 and

class probabilities 𝑞 𝑧 = 𝑗

• 𝑞 𝑧 = 1|𝑦 = 𝜏 𝑏 = 𝜏(𝑥𝑈𝑦 + 𝑐) is equivalent to setting log odds

𝑏 = ln 𝑞 𝑧 = 1|𝑦 𝑞 𝑧 = 2|𝑦 = 𝑥𝑈𝑦 + 𝑐

• Why linear log odds?

Review: binary logistic regression

• Suppose the class-conditional densities 𝑞 𝑦 𝑧 = 𝑗 is normal

𝑞 𝑦 𝑧 = 𝑗 = 𝑂 𝑦|𝜈𝑗, 𝐽 = 1 2𝜌 𝑒/2 exp{− 1 2 𝑦 − 𝜈𝑗

2}

• log odd is

𝑏 = ln 𝑞 𝑦|𝑧 = 1 𝑞(𝑧 = 1) 𝑞 𝑦|𝑧 = 2 𝑞(𝑧 = 2) = 𝑥𝑈𝑦 + 𝑐 where 𝑥 = 𝜈1 − 𝜈2, 𝑐 = − 1 2 𝜈1

𝑈𝜈1 + 1

2 𝜈2

𝑈𝜈2 + ln 𝑞(𝑧 = 1)

𝑞(𝑧 = 2)

Multiclass logistic regression

• Suppose we model the class-conditional densities 𝑞 𝑦 𝑧 = 𝑗 and

class probabilities 𝑞 𝑧 = 𝑗

• Conditional probability by Bayesian rule:

𝑞 𝑧 = 𝑗|𝑦 = 𝑞 𝑦|𝑧 = 𝑗 𝑞(𝑧 = 𝑗) σ𝑘 𝑞 𝑦|𝑧 = 𝑘 𝑞(𝑧 = 𝑘) = exp(𝑏𝑗) σ𝑘 exp(𝑏𝑘) where we define 𝑏𝑗 ≔ ln [𝑞 𝑦 𝑧 = 𝑗 𝑞 𝑧 = 𝑗 ]

Multiclass logistic regression

• Suppose the class-conditional densities 𝑞 𝑦 𝑧 = 𝑗 is normal

𝑞 𝑦 𝑧 = 𝑗 = 𝑂 𝑦|𝜈𝑗, 𝐽 = 1 2𝜌 𝑒/2 exp{− 1 2 𝑦 − 𝜈𝑗

2}

• Then

𝑏𝑗 ≔ ln 𝑞 𝑦 𝑧 = 𝑗 𝑞 𝑧 = 𝑗 = − 1 2 𝑦𝑈𝑦 + 𝑥𝑗

𝑈

𝑦 + 𝑐𝑗 where 𝑥𝑗 = 𝜈𝑗, 𝑐𝑗 = − 1 2 𝜈𝑗

𝑈𝜈𝑗 + ln 𝑞 𝑧 = 𝑗 + ln

1 2𝜌 𝑒/2

Multiclass logistic regression

• Suppose the class-conditional densities 𝑞 𝑦 𝑧 = 𝑗 is normal

𝑞 𝑦 𝑧 = 𝑗 = 𝑂 𝑦|𝜈𝑗, 𝐽 = 1 2𝜌 𝑒/2 exp{− 1 2 𝑦 − 𝜈𝑗

2}

• Cancel out −

1 2 𝑦𝑈𝑦, we have

𝑞 𝑧 = 𝑗|𝑦 = exp(𝑏𝑗) σ𝑘 exp(𝑏𝑘) , 𝑏𝑗 ≔ 𝑥𝑗 𝑈𝑦 + 𝑐𝑗 where 𝑥𝑗 = 𝜈𝑗, 𝑐𝑗 = − 1 2 𝜈𝑗

𝑈𝜈𝑗 + ln 𝑞 𝑧 = 𝑗 + ln

1 2𝜌 𝑒/2

Multiclass logistic regression: conclusion

• Suppose the class-conditional densities 𝑞 𝑦 𝑧 = 𝑗 is normal

𝑞 𝑦 𝑧 = 𝑗 = 𝑂 𝑦|𝜈𝑗, 𝐽 = 1 2𝜌 𝑒/2 exp{− 1 2 𝑦 − 𝜈𝑗

2}

• Then

𝑞 𝑧 = 𝑗|𝑦 = exp( 𝑥𝑗 𝑈𝑦 + 𝑐𝑗) σ𝑘 exp( 𝑥𝑘 𝑈𝑦 + 𝑐𝑘) which is the hypothesis class for multiclass logistic regression

• It is softmax on linear transformation; it can be used to derive the negative log-

likelihood loss (cross entropy)

Softmax

• A way to squash 𝑏 = (𝑏1, 𝑏2, … , 𝑏𝑗, … ) into probability vector 𝑞

softmax 𝑏 = exp(𝑏1) σ𝑘 exp(𝑏𝑘) , exp(𝑏2) σ𝑘 exp(𝑏𝑘) , … , exp 𝑏𝑗 σ𝑘 exp 𝑏𝑘 , …

• Behave like max: when 𝑏𝑗 ≫ 𝑏𝑘 ∀𝑘 ≠ 𝑗 , 𝑞𝑗 ≅ 1, 𝑞𝑘 ≅ 0

Cross entropy for conditional distribution

• Let 𝑞data(𝑧|𝑦) denote the empirical distribution of the data
• Negative log-likelihood

−

1 𝑜 σ𝑗=1 𝑜

log 𝑞 𝑧 = 𝑧𝑗 𝑦𝑗 = −E𝑞data(𝑧|𝑦) log 𝑞(𝑧|𝑦) is the cross entropy between 𝑞data and the model output 𝑞

• Information theory viewpoint: KL divergence

𝐸(𝑞data| 𝑞 = E𝑞data[log

𝑞data 𝑞 ] = E𝑞data [log 𝑞data] − E𝑞data[log 𝑞]

Entropy; constant Cross entropy

Cross entropy for full distribution

• Let 𝑞data(𝑦, 𝑧) denote the empirical distribution of the data
• Negative log-likelihood

−

1 𝑜 σ𝑗=1 𝑜

log 𝑞(𝑦𝑗, 𝑧𝑗) = −E𝑞data(𝑦,𝑧) log 𝑞(𝑦, 𝑧) is the cross entropy between 𝑞data and the model output 𝑞

Multiclass logistic regression: summary

Label 𝑧𝑗 (𝑥𝑘)𝑈ℎ + 𝑐𝑘 softmax Last hidden layer ℎ 𝑞𝑘 Cross entropy Linear Convert to probability Loss