optimization for machine learning
play

Optimization for Machine Learning Lecture 3: Bundle Methods S.V . - PowerPoint PPT Presentation

Sep 19, 2023 •148 likes •779 views

Optimization for Machine Learning Lecture 3: Bundle Methods S.V . N. (vishy) Vishwanathan Purdue University vishy@purdue.edu July 11, 2012 S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 1 / 30 Motivation

  1. Optimization for Machine Learning Lecture 3: Bundle Methods S.V . N. (vishy) Vishwanathan Purdue University vishy@purdue.edu July 11, 2012 S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 1 / 30

  2. Motivation Outline Motivation 1 Cutting Plane Methods 2 Non Smooth Functions 3 Bundle Methods 4 BMRM 5 Convergence Analysis 6 Experiments 7 Lower Bounds 8 References 9 S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 2 / 30

  3. Motivation Regularized Risk Minimization Objective Function Training data: { x 1 , . . . , x m } Labels: { y 1 , . . . , y m } Learn a vector: w m + 1 � J ( w ) := λ Ω( w ) l ( x i , y i , w ) minimize m w � �� � i =1 Regularizer � �� � Risk R emp S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 3 / 30

  4. Motivation Binary Classification y i = +1 y i = − 1 S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 4 / 30

  5. Motivation Binary Classification y i = +1 y i = − 1 S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 4 / 30

  6. Motivation Binary Classification y i = +1 y i = − 1 S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 4 / 30

  7. Motivation Binary Classification � w , x 1 � + b = +1 y i = +1 � w , x 2 � + b = − 1 � w , x 1 − x 2 � = 2 � � 2 w � w � , x 1 − x 2 = � w � x 2 x 1 y i = − 1 { x | � w , x � + b = 1 } { x | � w , x � + b = − 1 } { x | � w , x � + b = 0 } S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 4 / 30

  8. Motivation Linear Support Vector Machines Optimization Problem 2 max � w � w , b s.t. y i ( � w , x i � + b ) ≥ 1 for all i S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 5 / 30

  9. Motivation Linear Support Vector Machines Optimization Problem 1 2 � w � 2 min w , b s.t. y i ( � w , x i � + b ) ≥ 1 for all i S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 5 / 30

  10. Motivation Linear Support Vector Machines Optimization Problem 1 2 � w � 2 min w , b ,ξ s.t. y i ( � w , x i � + b ) ≥ 1 − ξ i for all i ξ i ≥ 0 S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 5 / 30

  11. Motivation Linear Support Vector Machines Optimization Problem m 2 � w � 2 + 1 λ � min ξ i m w , b ,ξ i =1 s.t. y i ( � w , x i � + b ) ≥ 1 − ξ i for all i ξ i ≥ 0 S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 5 / 30

  12. Motivation Linear Support Vector Machines Optimization Problem m 2 � w � 2 + 1 λ � min ξ i m w , b ,ξ i =1 s.t. ξ i ≥ 1 − y i ( � w , x i � + b ) for all i ξ i ≥ 0 S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 5 / 30

  13. Motivation Linear Support Vector Machines Optimization Problem m λ 2 � w � 2 + 1 � min max(0 , 1 − y i ( � w , x i � + b )) m w , b i =1 S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 5 / 30

  14. Motivation Linear Support Vector Machines Optimization Problem m λ + 1 � 2 � w � 2 min max(0 , 1 − y i ( � w , x i � + b )) m w , b i =1 � �� � � �� � λ Ω( w ) R emp ( w ) S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 5 / 30

  15. Motivation Binary Hinge Loss loss y ( � w , x � + b ) S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 6 / 30

  16. Cutting Plane Methods Outline Motivation 1 Cutting Plane Methods 2 Non Smooth Functions 3 Bundle Methods 4 BMRM 5 Convergence Analysis 6 Experiments 7 Lower Bounds 8 References 9 S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 7 / 30

  17. Cutting Plane Methods First Order Taylor Expansion The First Order Taylor approximation globally lower bounds the function For any x and x ′ we have � � f ( x ) ≥ f ( x ′ ) + x − x ′ , ∇ f ( x ′ ) S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 8 / 30

  18. Cutting Plane Methods Cutting Plane Methods S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 9 / 30

  19. Cutting Plane Methods Cutting Plane Methods S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 9 / 30

  20. Cutting Plane Methods Cutting Plane Methods S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 9 / 30

  21. Cutting Plane Methods Cutting Plane Methods S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 9 / 30

  22. Cutting Plane Methods Cutting Plane Methods S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 9 / 30

  23. Cutting Plane Methods Cutting Plane Methods S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 9 / 30

  24. Cutting Plane Methods Cutting Plane Methods S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 9 / 30

  25. Cutting Plane Methods Cutting Plane Methods S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 9 / 30

  26. Cutting Plane Methods Cutting Plane Methods S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 9 / 30

  27. Cutting Plane Methods In a Nutshell Cutting Plane Methods work by forming the piecewise linear lower bound J ( w ) ≥ J CP ( w ) := max 1 ≤ i ≤ t { J ( w i − 1 ) + � w − w i − 1 , s i �} . t where s i denotes the gradient ∇ J ( w i − 1 ). At iteration t the set { w i } t − 1 i =0 is augmented by J CP w t := argmin ( w ) . t w Stop when the duality gap 0 ≤ i ≤ t J ( w i ) − J CP ǫ t := min ( w t ) t falls below a pre-specified threshold ǫ . S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 10 / 30

  28. Cutting Plane Methods In a Nutshell Cutting Plane Methods work by forming the piecewise linear lower bound J ( w ) ≥ J CP ( w ) := max 1 ≤ i ≤ t { J ( w i − 1 ) + � w − w i − 1 , s i �} . t where s i denotes the gradient ∇ J ( w i − 1 ). At iteration t the set { w i } t − 1 i =0 is augmented by J CP w t := argmin ( w ) . t w Stop when the duality gap 0 ≤ i ≤ t J ( w i ) − J CP ǫ t := min ( w t ) t falls below a pre-specified threshold ǫ . S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 10 / 30

  29. Cutting Plane Methods In a Nutshell Cutting Plane Methods work by forming the piecewise linear lower bound J ( w ) ≥ J CP ( w ) := max 1 ≤ i ≤ t { J ( w i − 1 ) + � w − w i − 1 , s i �} . t where s i denotes the gradient ∇ J ( w i − 1 ). At iteration t the set { w i } t − 1 i =0 is augmented by J CP w t := argmin ( w ) . t w Stop when the duality gap 0 ≤ i ≤ t J ( w i ) − J CP ǫ t := min ( w t ) t falls below a pre-specified threshold ǫ . S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 10 / 30

  30. Non Smooth Functions Outline Motivation 1 Cutting Plane Methods 2 Non Smooth Functions 3 Bundle Methods 4 BMRM 5 Convergence Analysis 6 Experiments 7 Lower Bounds 8 References 9 S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 11 / 30

  31. Non Smooth Functions What if the Function is NonSmooth? The piecewise linear function J ( w ) := max � u i , w � i is convex but not differentiable at the kinks! S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 12 / 30

  32. Non Smooth Functions Subgradients to the Rescue A subgradient at w ′ is any vector s which satisfies � � J ( w ) ≥ J ( w ′ ) + w − w ′ , s for all w Set of all subgradients is denoted as ∂ J ( w ) S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 13 / 30

  33. Non Smooth Functions Subgradients to the Rescue A subgradient at w ′ is any vector s which satisfies � � J ( w ) ≥ J ( w ′ ) + w − w ′ , s for all w Set of all subgradients is denoted as ∂ J ( w ) S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 13 / 30

  34. Non Smooth Functions Subgradients to the Rescue A subgradient at w ′ is any vector s which satisfies � � J ( w ) ≥ J ( w ′ ) + w − w ′ , s for all w Set of all subgradients is denoted as ∂ J ( w ) S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 13 / 30

  35. Non Smooth Functions Good News! Cutting Plane Methods work with subgradients Just choose an arbitrary one S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 14 / 30

  36. Non Smooth Functions Good News! Cutting Plane Methods work with subgradients Just choose an arbitrary one Then what is the bad news? S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 14 / 30

  37. Non Smooth Functions Bad News 1 0 . 8 0 . 6 1 0 . 5 0 − 1 − 0 . 5 0 − 0 . 5 0 . 5 1 − 1 S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 15 / 30

  38. Bundle Methods Outline Motivation 1 Cutting Plane Methods 2 Non Smooth Functions 3 Bundle Methods 4 BMRM 5 Convergence Analysis 6 Experiments 7 Lower Bounds 8 References 9 S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 16 / 30

Recommend

Machine Learning for Auto Optimization What is Machine Learning? Definition: Machine

Machine Learning for Auto Optimization What is Machine Learning? Definition: Machine

Machine Learning for Auto Optimization What is Machine Learning? Definition: Machine learning refers to any system where the performance of a machine in performing a task improves by gaining more experience in performing that task .

394 views • 27 slides
Outline Optimization Unconstrained Optimization Problems Machine Learning and Pattern

Outline Optimization Unconstrained Optimization Problems Machine Learning and Pattern

Outline Optimization Unconstrained Optimization Problems Machine Learning and Pattern Recognition Gradient descent Second order methods Chris Williams Constrained Optimization Problems Linear programming School of

540 views • 8 slides
Bayesian Optimization CSC2541 - Topics in Machine Learning Scalable and Flexible Models of

Bayesian Optimization CSC2541 - Topics in Machine Learning Scalable and Flexible Models of

Bayesian Optimization CSC2541 - Topics in Machine Learning Scalable and Flexible Models of Uncertainty University of Toronto - Fall 2017 Overview 1. Bayesian Optimization of Machine Learning Algorithms 2. Gaussian Process Optimization in the

1.53k views • 94 slides
Short Course in Supervised Learning Robust Optimization and Machine Learning Robust Supervised

Short Course in Supervised Learning Robust Optimization and Machine Learning Robust Supervised

Robust Optimization & Machine Learning 6. Robust Optimization Short Course in Supervised Learning Robust Optimization and Machine Learning Robust Supervised Learning Motivations Examples Thresholding and robustness Boolean data

724 views • 48 slides
Machine learning and black-box expensive optimization S ebastien Verel Laboratoire

Machine learning and black-box expensive optimization S ebastien Verel Laboratoire

Introduction Learning for optimization Optimization for learning Machine learning and black-box expensive optimization S ebastien Verel Laboratoire dInformatique, Signal et Image de la C ote dopale (LISIC) Universit e du

305 views • 19 slides
Outline IAML: Optimization Why we use optimization in machine learning The general

Outline IAML: Optimization Why we use optimization in machine learning The general

Outline IAML: Optimization Why we use optimization in machine learning The general optimization problem Gradient descent Nigel Goddard Problems with gradient descent School of Informatics Batch versus online Second-order

333 views • 6 slides
Local Function Optimization COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning

Local Function Optimization COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning

Local Function Optimization COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Local Function Optimization 1 / 29 Outline 1 Gradient, Hessian, and Convexity 2 A Local, Unconstrained Optimization Template 3 Steepest Descent 4

629 views • 29 slides
Convex Optimization by Stephen Boyd, and Lieven Vandenberghe. Optimization for Machine Learning by

Convex Optimization by Stephen Boyd, and Lieven Vandenberghe. Optimization for Machine Learning by

Convex Optimization by Stephen Boyd, and Lieven Vandenberghe. Optimization for Machine Learning by Suvrit Sra, Sebastian Nowozin, and Stephen J. Wright Convex Optimization- Chapter 1: Introduction mathematical optimization Least-squares and

764 views • 20 slides
Bayesian Optimization and Automated Machine Learning Jungtaek Kim (jtkim@postech.ac.kr) Machine

Bayesian Optimization and Automated Machine Learning Jungtaek Kim (jtkim@postech.ac.kr) Machine

Bayesian Optimization and Automated Machine Learning Jungtaek Kim (jtkim@postech.ac.kr) Machine Learning Group, Department of Computer Science and Engineering, POSTECH, 77 Cheongam-ro, Nam-gu, Pohang 37673, Gyeongsangbuk-do, Republic of Korea

961 views • 28 slides
Harnessing Structure in Optimization for Machine Learning Franck Iutzeler LJK, Univ. Grenoble

Harnessing Structure in Optimization for Machine Learning Franck Iutzeler LJK, Univ. Grenoble

Harnessing Structure in Optimization for Machine Learning Franck Iutzeler LJK, Univ. Grenoble Alpes Optimization for Machine Learning CIRM 9-13 March 2020 >>> Regularization in Learning Structure Regularization Linear inverse

507 views • 33 slides
Scalable Machine Learning 4. Optimization Alex Smola Yahoo! Research and ANU

Scalable Machine Learning 4. Optimization Alex Smola Yahoo! Research and ANU

Scalable Machine Learning 4. Optimization Alex Smola Yahoo! Research and ANU http://alex.smola.org/teaching/berkeley2012 Stat 260 SP 12 4. Optimization Optimization Basic Techniques Gradient descent Newton's method Conjugate

1.86k views • 133 slides
Optimization and Machine Learning with Applications Antonio Candelieri 1,2 Department of Computer

Optimization and Machine Learning with Applications Antonio Candelieri 1,2 Department of Computer

Optimization and Machine Learning with Applications Antonio Candelieri 1,2 Department of Computer Science, Systems and Communications University of Milano-Bicocca, viale Sarca 336, 20126, Milan, Italy OAKS srl Optimization Analytics Knowledge

515 views • 26 slides
Optimization for Machine Learning Tom Schaul schaul@cims.nyu.edu Recap: Learning Machines

Optimization for Machine Learning Tom Schaul schaul@cims.nyu.edu Recap: Learning Machines

Optimization for Machine Learning Tom Schaul schaul@cims.nyu.edu Recap: Learning Machines Learning machines (Neural Networks, etc.) Forward passes produce a function of input Trainable parameters (aka weights, biases, etc.)

815 views • 50 slides
Stochastic Optimization Techniques for Big Data Machine Learning Tong Zhang Rutgers University

Stochastic Optimization Techniques for Big Data Machine Learning Tong Zhang Rutgers University

Stochastic Optimization Techniques for Big Data Machine Learning Tong Zhang Rutgers University & Baidu Inc. T. Zhang Big Data Optimization 1 / 73 Outline Background: big data optimization in machine learning: special structure T. Zhang

1.95k views • 129 slides
Mitigating the Compiler Optimization Phase-Ordering Problem using Machine Learning Sameer

Mitigating the Compiler Optimization Phase-Ordering Problem using Machine Learning Sameer

Mitigating the Compiler Optimization Phase-Ordering Problem using Machine Learning Sameer Kulkarni & John Cavazos Intro - Why we need code optimization? - Prog. Language design flaw - e.g. goto statement in C/Cpp - People are evil and

854 views • 19 slides
Dynamic gradient estimation in machine learning Thomas Flynn Abstract The optimization problems

Dynamic gradient estimation in machine learning Thomas Flynn Abstract The optimization problems

Dynamic gradient estimation in machine learning Thomas Flynn Abstract The optimization problems arising in machine learning form some of the most theoret- ically challenging and computationally demanding problems in numerical computing today.

593 views • 45 slides
Machine Learning and Optimization Alessio Signorini <alessio.signorini@oneriot.com>

Machine Learning and Optimization Alessio Signorini <alessio.signorini@oneriot.com>

Practical Introduction to Machine Learning and Optimization Alessio Signorini <alessio.signorini@oneriot.com> Everyday's Optimizations Although you may not know, everybody uses daily some sort of optimization technique: Timing your

502 views • 34 slides
Optimization for Machine Learning Lecture 2: Support Vector Machine Training S.V . N. (vishy)

Optimization for Machine Learning Lecture 2: Support Vector Machine Training S.V . N. (vishy)

Optimization for Machine Learning Lecture 2: Support Vector Machine Training S.V . N. (vishy) Vishwanathan Purdue University vishy@purdue.edu July 11, 2012 S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 1 / 41

835 views • 69 slides
Advanced Machine Learning CS 7140 - Spring 2019 Lecture 24: Bayesian Optimization Jan-Willem van

Advanced Machine Learning CS 7140 - Spring 2019 Lecture 24: Bayesian Optimization Jan-Willem van

Advanced Machine Learning CS 7140 - Spring 2019 Lecture 24: Bayesian Optimization Jan-Willem van de Meent Slide credits: Ryan Adams, Nando de Freitas Background: Multi-Armed Bandits Problem: Which machine has highest rate of payout?

805 views • 42 slides
Introduction to Convex Optimization for Machine Learning John Duchi University of California,

Introduction to Convex Optimization for Machine Learning John Duchi University of California,

Introduction to Convex Optimization for Machine Learning John Duchi University of California, Berkeley Practical Machine Learning, Fall 2009 Duchi (UC Berkeley) Convex Optimization for Machine Learning Fall 2009 1 / 53 Outline What is

838 views • 82 slides
Optimization in the Big Data Regime Sham M. Kakade Machine Learning for Big Data

Optimization in the Big Data Regime Sham M. Kakade Machine Learning for Big Data

Optimization in the Big Data Regime Sham M. Kakade Machine Learning for Big Data CSE547/STAT548 University of Washington S. M. Kakade (UW) Optimization for Big data 1 / 18 Announcements... HW2 due Mon. Work on your project milestones

833 views • 21 slides
Optimization for Machine Learning Lecture 4: SMO-MKL S.V . N. (vishy) Vishwanathan Purdue

Optimization for Machine Learning Lecture 4: SMO-MKL S.V . N. (vishy) Vishwanathan Purdue

Optimization for Machine Learning Lecture 4: SMO-MKL S.V . N. (vishy) Vishwanathan Purdue University vishy@purdue.edu July 11, 2012 S.V . N. Vishwanathan (Purdue University) Optimization for Machine Learning 1 / 22 Motivation Binary

720 views • 39 slides
Improving Optimization Bounds using Machine Learning: Decision Diagrams meet Deep Reinforcement

Improving Optimization Bounds using Machine Learning: Decision Diagrams meet Deep Reinforcement

Improving Optimization Bounds using Machine Learning: Decision Diagrams meet Deep Reinforcement Learning Quentin Cappart , Emmanuel Goutierre, David Bergman, Louis-Martin Rousseau 1 Research question Bounding mechanisms are critical in the

449 views • 24 slides
BBM406 Fundamentals of Machine Learning Lecture 4: Linear Regression, Optimization,

BBM406 Fundamentals of Machine Learning Lecture 4: Linear Regression, Optimization,

illustration: detail from xkcd strip #2048 BBM406 Fundamentals of Machine Learning Lecture 4: Linear Regression, Optimization, Generalization, Model complexity, Regularization Aykut Erdem // Hacettepe University // Fall 2019 1 ,

1.09k views • 56 slides

More recommend