Lecture 5: SVM II Princeton University COS 495 Instructor: Yingyu - - PowerPoint PPT Presentation

Machine Learning Basics Lecture 5: SVM II

Princeton University COS 495 Instructor: Yingyu Liang

Review: SVM objective

SVM: objective

• Let 𝑧𝑗 ∈ +1, −1 , 𝑔

𝑥,𝑐 𝑦 = 𝑥𝑈𝑦 + 𝑐. Margin:

𝛿 = min

𝑗

𝑧𝑗𝑔

𝑥,𝑐 𝑦𝑗

| 𝑥 |

• Support Vector Machine:

max

𝑥,𝑐 𝛿 = max 𝑥,𝑐 min 𝑗

𝑧𝑗𝑔

𝑥,𝑐 𝑦𝑗

| 𝑥 |

SVM: optimization

• Optimization (Quadratic Programming):

min

𝑥,𝑐

1 2 𝑥

2

𝑧𝑗 𝑥𝑈𝑦𝑗 + 𝑐 ≥ 1, ∀𝑗

• Solved by Lagrange multiplier method:

ℒ 𝑥, 𝑐, 𝜷 = 1 2 𝑥

2

− ෍

𝑗

𝛽𝑗[𝑧𝑗 𝑥𝑈𝑦𝑗 + 𝑐 − 1] where 𝜷 is the Lagrange multiplier

Lagrange multiplier

Lagrangian

• Consider optimization problem:

min

𝑥

𝑔(𝑥) ℎ𝑗 𝑥 = 0, ∀1 ≤ 𝑗 ≤ 𝑚

• Lagrangian:

ℒ 𝑥, 𝜸 = 𝑔 𝑥 + ෍

𝑗

𝛾𝑗ℎ𝑗(𝑥) where 𝛾𝑗’s are called Lagrange multipliers

Lagrangian

• Consider optimization problem:

min

𝑥

𝑔(𝑥) ℎ𝑗 𝑥 = 0, ∀1 ≤ 𝑗 ≤ 𝑚

• Solved by setting derivatives of Lagrangian to 0

𝜖ℒ 𝜖𝑥𝑗 = 0; 𝜖ℒ 𝜖𝛾𝑗 = 0

Generalized Lagrangian

• Consider optimization problem:

min

𝑥

𝑔(𝑥) 𝑕𝑗 𝑥 ≤ 0, ∀1 ≤ 𝑗 ≤ 𝑙 ℎ𝑘 𝑥 = 0, ∀1 ≤ 𝑘 ≤ 𝑚

• Generalized Lagrangian:

ℒ 𝑥, 𝜷, 𝜸 = 𝑔 𝑥 + ෍

𝑗

𝛽𝑗𝑕𝑗(𝑥) + ෍

𝑘

𝛾𝑘ℎ𝑘(𝑥) where 𝛽𝑗, 𝛾𝑘’s are called Lagrange multipliers

Generalized Lagrangian

• Consider the quantity:

𝜄𝑄 𝑥 ≔ max

𝜷,𝜸:𝛽𝑗≥0 ℒ 𝑥, 𝜷, 𝜸

• Why?

𝜄𝑄 𝑥 = ቊ𝑔 𝑥 , if 𝑥 satisfies all the constraints +∞, if 𝑥 does not satisfy the constraints

• So minimizing 𝑔 𝑥 is the same as minimizing 𝜄𝑄 𝑥

min

𝑥 𝑔 𝑥 = min 𝑥 𝜄𝑄 𝑥 = min 𝑥

max

𝜷,𝜸:𝛽𝑗≥0 ℒ 𝑥, 𝜷, 𝜸

Lagrange duality

• The primal problem

𝑞∗ ≔ min

𝑥 𝑔 𝑥 = min 𝑥

max

𝜷,𝜸:𝛽𝑗≥0 ℒ 𝑥, 𝜷, 𝜸

• The dual problem

𝑒∗ ≔ max

𝜷,𝜸:𝛽𝑗≥0min 𝑥 ℒ 𝑥, 𝜷, 𝜸

• Always true:

𝑒∗ ≤ 𝑞∗

Lagrange duality

• The primal problem

𝑞∗ ≔ min

𝑥 𝑔 𝑥 = min 𝑥

max

𝜷,𝜸:𝛽𝑗≥0 ℒ 𝑥, 𝜷, 𝜸

• The dual problem

𝑒∗ ≔ max

𝜷,𝜸:𝛽𝑗≥0min 𝑥 ℒ 𝑥, 𝜷, 𝜸

• Interesting case: when do we have

𝑒∗ = 𝑞∗?

Lagrange duality

• Theorem: under proper conditions, there exists 𝑥∗, 𝜷∗, 𝜸∗ such that

𝑒∗ = ℒ 𝑥∗, 𝜷∗, 𝜸∗ = 𝑞∗ Moreover, 𝑥∗, 𝜷∗, 𝜸∗ satisfy Karush-Kuhn-Tucker (KKT) conditions: 𝜖ℒ 𝜖𝑥𝑗 = 0, 𝛽𝑗𝑕𝑗 𝑥 = 0 𝑕𝑗 𝑥 ≤ 0, ℎ𝑘 𝑥 = 0, 𝛽𝑗 ≥ 0

Lagrange duality

• Theorem: under proper conditions, there exists 𝑥∗, 𝜷∗, 𝜸∗ such that

𝑒∗ = ℒ 𝑥∗, 𝜷∗, 𝜸∗ = 𝑞∗ Moreover, 𝑥∗, 𝜷∗, 𝜸∗ satisfy Karush-Kuhn-Tucker (KKT) conditions: 𝜖ℒ 𝜖𝑥𝑗 = 0, 𝛽𝑗𝑕𝑗 𝑥 = 0 𝑕𝑗 𝑥 ≤ 0, ℎ𝑘 𝑥 = 0, 𝛽𝑗 ≥ 0 dual complementarity

Lagrange duality

• Theorem: under proper conditions, there exists 𝑥∗, 𝜷∗, 𝜸∗ such that

𝑒∗ = ℒ 𝑥∗, 𝜷∗, 𝜸∗ = 𝑞∗

• Moreover, 𝑥∗, 𝜷∗, 𝜸∗ satisfy Karush-Kuhn-Tucker (KKT) conditions:

𝜖ℒ 𝜖𝑥𝑗 = 0, 𝛽𝑗𝑕𝑗 𝑥 = 0 𝑕𝑗 𝑥 ≤ 0, ℎ𝑘 𝑥 = 0, 𝛽𝑗 ≥ 0 dual constraints primal constraints

Lagrange duality

• What are the proper conditions?
• A set of conditions (Slater conditions):
• 𝑔, 𝑕𝑗 convex, ℎ𝑘 affine
• Exists 𝑥 satisfying all 𝑕𝑗 𝑥 < 0
• There exist other sets of conditions
• Search Karush–Kuhn–Tucker conditions on Wikipedia

SVM: optimization

SVM: optimization

• Optimization (Quadratic Programming):

min

𝑥,𝑐

1 2 𝑥

2

𝑧𝑗 𝑥𝑈𝑦𝑗 + 𝑐 ≥ 1, ∀𝑗

• Generalized Lagrangian:

ℒ 𝑥, 𝑐, 𝜷 = 1 2 𝑥

2

− ෍

𝑗

𝛽𝑗[𝑧𝑗 𝑥𝑈𝑦𝑗 + 𝑐 − 1] where 𝜷 is the Lagrange multiplier

SVM: optimization

• KKT conditions:

𝜖ℒ 𝜖𝑥 = 0,  𝑥 = σ𝑗 𝛽𝑗𝑧𝑗𝑦𝑗 (1) 𝜖ℒ 𝜖𝑐 = 0,  0 = σ𝑗 𝛽𝑗𝑧𝑗

(2)

• Plug into ℒ:

ℒ 𝑥, 𝑐, 𝜷 = σ𝑗 𝛽𝑗 −

1 2 σ𝑗𝑘 𝛽𝑗𝛽𝑘𝑧𝑗𝑧𝑘𝑦𝑗 𝑈𝑦𝑘 (3)

combined with 0 = σ𝑗 𝛽𝑗𝑧𝑗 , 𝛽𝑗 ≥ 0

SVM: optimization

• Reduces to dual problem:

ℒ 𝑥, 𝑐, 𝜷 = ෍

𝑗

𝛽𝑗 − 1 2 ෍

𝑗𝑘

𝛽𝑗𝛽𝑘𝑧𝑗𝑧𝑘𝑦𝑗

𝑈𝑦𝑘

෍

𝑗

𝛽𝑗𝑧𝑗 = 0, 𝛽𝑗 ≥ 0

• Since 𝑥 = σ𝑗 𝛽𝑗𝑧𝑗𝑦𝑗, we have 𝑥𝑈𝑦 + 𝑐 = σ𝑗 𝛽𝑗𝑧𝑗𝑦𝑗

𝑈𝑦 + 𝑐

Only depend on inner products

Kernel methods

Features

Color Histogram

Red Green Blue

Extract features

𝑦 𝜚 𝑦

Features

Features

• Proper feature mapping can make non-linear to linear
• Using SVM on the feature space {𝜚 𝑦𝑗 }: only need 𝜚 𝑦𝑗 𝑈𝜚(𝑦𝑘)
• Conclusion: no need to design 𝜚 ⋅ , only need to design

𝑙 𝑦𝑗, 𝑦𝑘 = 𝜚 𝑦𝑗 𝑈𝜚(𝑦𝑘)

Polynomial kernels

• Fix degree 𝑒 and constant 𝑑:

𝑙 𝑦, 𝑦′ = 𝑦𝑈𝑦′ + 𝑑 𝑒

• What are 𝜚(𝑦)?
• Expand the expression to get 𝜚(𝑦)

Polynomial kernels

Figure from Foundations of Machine Learning, by M. Mohri, A. Rostamizadeh, and A. Talwalkar

Figure from Foundations of Machine Learning, by M. Mohri, A. Rostamizadeh, and A. Talwalkar

Gaussian kernels

• Fix bandwidth 𝜏:

𝑙 𝑦, 𝑦′ = exp(− 𝑦 − 𝑦′

2/2𝜏2)

• Also called radial basis function (RBF) kernels
• What are 𝜚(𝑦)? Consider the un-normalized version

𝑙′ 𝑦, 𝑦′ = exp(𝑦𝑈𝑦′/𝜏2)

• Power series expansion:

𝑙′ 𝑦, 𝑦′ = ෍

𝑗 +∞ 𝑦𝑈𝑦′ 𝑗

𝜏𝑗𝑗!

Mercer’s condition for kenerls

• Theorem: 𝑙 𝑦, 𝑦′ has expansion

𝑙 𝑦, 𝑦′ = ෍

𝑗 +∞

𝑏𝑗𝜚𝑗 𝑦 𝜚𝑗(𝑦′) if and only if for any function 𝑑(𝑦), ∫ ∫ 𝑑 𝑦 𝑑 𝑦′ 𝑙 𝑦, 𝑦′ 𝑒𝑦𝑒𝑦′ ≥ 0 (Omit some math conditions for 𝑙 and 𝑑)

Constructing new kernels

• Kernels are closed under positive scaling, sum, product, pointwise

limit, and composition with a power series σ𝑗

+∞ 𝑏𝑗𝑙𝑗(𝑦, 𝑦′)

• Example: 𝑙1 𝑦, 𝑦′ , 𝑙2 𝑦, 𝑦′ are kernels, then also is

𝑙 𝑦, 𝑦′ = 2𝑙1 𝑦, 𝑦′ + 3𝑙2 𝑦, 𝑦′

• Example: 𝑙1 𝑦, 𝑦′ is kernel, then also is

𝑙 𝑦, 𝑦′ = exp(𝑙1 𝑦, 𝑦′ )

Kernels v.s. Neural networks

Features

Color Histogram

Red Green Blue

Extract features

𝑦 𝑧 = 𝑥𝑈𝜚 𝑦

build hypothesis

Features: part of the model

𝑧 = 𝑥𝑈𝜚 𝑦

build hypothesis

Linear model Nonlinear model

Polynomial kernels

Figure from Foundations of Machine Learning, by M. Mohri, A. Rostamizadeh, and A. Talwalkar

Polynomial kernel SVM as two layer neural network

𝑦1 𝑦2 𝑦1

2

𝑦2

2

2𝑦1𝑦2 2𝑑𝑦1 2𝑑𝑦2 𝑑 𝑧 = sign(𝑥𝑈𝜚(𝑦) + 𝑐) First layer is fixed. If also learn first layer, it becomes two layer neural network