optimization introduction optimization
play

Optimization (Introduction) Optimization Goal: Find the minimizer - PowerPoint PPT Presentation

Jan 04, 2023 β€’277 likes β€’534 views

Optimization (Introduction) Optimization Goal: Find the minimizer that minimizes the objective (cost) function : " Unconstrained Optimization Optimization Goal: Find the minimizer that minimizes the

  1. Optimization (Introduction)

  2. Optimization Goal: Find the minimizer π’š βˆ— that minimizes the objective (cost) function 𝑔 π’š : β„› " β†’ β„› Unconstrained Optimization

  3. Optimization Goal: Find the minimizer π’š βˆ— that minimizes the objective (cost) function 𝑔 π’š : β„› " β†’ β„› Constrained Optimization

  4. Unconstrained Optimization β€’ What if we are looking for a maximizer π’š βˆ— ? 𝑔 π’š βˆ— = max 𝑔 π’š π’š

  5. Calculus problem: maximize the rectangle area subject to perimeter constraint max 𝒆 ∈ β„› !

  6. 𝐡𝑠𝑓𝑏 = 𝑒 $ 𝑒 % 𝑒 % 𝑒 $ 𝑄𝑓𝑠𝑗𝑛𝑓𝑒𝑓𝑠 = 2(𝑒 $ + 𝑒 % ) 𝑒 % 𝑒 $

  8. Unconstrained Optimization 1D

  9. What is the optimal solution? (1D) 𝑔 𝑦 βˆ— = min 𝑔 𝑦 " (First-order) Necessary condition (Second-order) Sufficient condition

  10. Types of optimization problems 𝑔 𝑦 βˆ— = min 𝑔: nonlinear, continuous 𝑔 𝑦 and smooth " Gradient-free methods Evaluate 𝑔 𝑦 Gradient (first-derivative) methods Evaluate 𝑔 𝑦 , 𝑔′ 𝑦 Second-derivative methods Evaluate 𝑔 𝑦 , 𝑔′ 𝑦 , 𝑔′′ 𝑦

  11. Does the solution exists? Local or global solution?

  12. Example (1D) Consider the function 𝑔 π’š = 7 ! 8 βˆ’ 7 " 9 βˆ’ 11 𝑦 : + 40𝑦 . Find the stationary point and check the sufficient condition 100 - 6 - 4 - 2 2 4 6 - 100 - 200

  13. Optimization in 1D: Golden Section Search β€’ Similar idea of bisection method for root finding β€’ Needs to bracket the minimum inside an interval β€’ Required the function to be unimodal A function 𝑔: β„› β†’ β„› is unimodal on an interval [𝑏, 𝑐] ΓΌ There is a unique π’š βˆ— ∈ [𝑏, 𝑐] such that 𝑔(π’š βˆ— ) is the minimum in [𝑏, 𝑐] ΓΌ For any 𝑦 ; , 𝑦 : ∈ [𝑏, 𝑐] with 𝑦 ; < 𝑦 : 𝑦 : < π’š βˆ— ⟹ 𝑔(𝑦 ; ) > 𝑔(𝑦 : ) Β§ 𝑦 ; > π’š βˆ— ⟹ 𝑔(𝑦 ; ) < 𝑔(𝑦 : ) Β§

  14. 𝑔 𝑔 $ $ 𝑔 𝑔 & & 𝑔 ' 𝑔 𝑔 % % 𝑔 ' 𝑦 $ 𝑦 $ 𝑦 & 𝑦 & 𝑦 % 𝑦 ' 𝑦 % 𝑦 ' 𝑑 𝑑 𝑏 𝑏 𝑐 𝑐

  17. Golden Section Search

  18. Golden Section Search What happens with the length of the interval after one iteration? β„Ž ! = 𝜐 β„Ž " Or in general: β„Ž #$! = 𝜐 β„Ž # Hence the interval gets reduced by 𝝊 (for bisection method to solve nonlinear equations, 𝜐 =0.5) For recursion: 𝜐 β„Ž ! = (1 βˆ’ 𝜐) β„Ž " 𝜐 𝜐 β„Ž " = (1 βˆ’ 𝜐) β„Ž " 𝜐 % = (1 βˆ’ 𝜐) 𝝊 = 𝟏. πŸ•πŸπŸ—

  19. Golden Section Search β€’ Derivative free method! β€’ Slow convergence: |𝑓 ?B; | lim = 0.618 𝑠 = 1 (π‘šπ‘—π‘œπ‘“π‘π‘  π‘‘π‘π‘œπ‘€π‘“π‘ π‘•π‘“π‘œπ‘‘π‘“) 𝑓 ? ?β†’A β€’ Only one function evaluation per iteration

  20. Example

  21. Newton’s Method Using Taylor Expansion, we can approximate the function 𝑔 with a quadratic function about 𝑦 C 𝑔 𝑦 β‰ˆ 𝑔 𝑦 C + 𝑔 D 𝑦 C (𝑦 βˆ’ 𝑦 C ) + ; : 𝑔 D β€² 𝑦 C (𝑦 βˆ’ 𝑦 C ) : And we want to find the minimum of the quadratic function using the first-order necessary condition

  22. Newton’s Method β€’ Algorithm: 𝑦 0 = starting guess 𝑦 123 = 𝑦 1 βˆ’ 𝑔′ 𝑦 1 /𝑔′′ 𝑦 1 β€’ Convergence: β€’ Typical quadratic convergence β€’ Local convergence (start guess close to solution) β€’ May fail to converge, or converge to a maximum or point of inflection

  23. Newton’s Method (Graphical Representation)

  24. Example Consider the function 𝑔 𝑦 = 4 𝑦 9 + 2 𝑦 : + 5 𝑦 + 40 If we use the initial guess 𝑦 C = 2 , what would be the value of 𝑦 after one iteration of the Newton’s method?

Recommend

15-780: Optimization J. Zico Kolter March 14-16, 2015 1 Outline Introduction to optimization

15-780: Optimization J. Zico Kolter March 14-16, 2015 1 Outline Introduction to optimization

15-780: Optimization J. Zico Kolter March 14-16, 2015 1 Outline Introduction to optimization Types of optimization problems Unconstrained optimization Constrained optimization Practical optimization 2 Outline Introduction to optimization

603 views β€’ 59 slides
Optimization (Introduction) : IR IR f ( x ) ID Optimization &quot; 112 FCI ) : IR NE

Optimization (Introduction) : IR IR f ( x ) ID Optimization &quot; 112 FCI ) : IR NE

Optimization (Introduction) : IR IR f ( x ) ID Optimization &quot; 112 FCI ) : IR NE Goal: Find the minimizer ! that minimizes the objective (cost) function &quot; ! : &quot; - fatal * Unconstrained Optimization x :

408 views β€’ 23 slides
Introduction: Why Optimization? Geoff Gordon &amp; Ryan Tibshirani Optimization 10-725 / 36-725

Introduction: Why Optimization? Geoff Gordon &amp; Ryan Tibshirani Optimization 10-725 / 36-725

Introduction: Why Optimization? Geoff Gordon &amp; Ryan Tibshirani Optimization 10-725 / 36-725 1 Where this course fits in In many ML/statistics/engineering courses, you learn how to: translate into min f ( x ) Question/idea Optimization

138 views β€’ 9 slides
Introduction to Optimization Dr. Mihail October 23, 2018 (Dr. Mihail) Optimization October 23,

Introduction to Optimization Dr. Mihail October 23, 2018 (Dr. Mihail) Optimization October 23,

Introduction to Optimization Dr. Mihail October 23, 2018 (Dr. Mihail) Optimization October 23, 2018 1 / 20 Overview What is optimization? Optimization is a mathematical discipline concerned with finding the maxima and minima of functions,

476 views β€’ 21 slides
Introduction: Mathematical optimization Motivating Example Applications Least-squares(LS) and

Introduction: Mathematical optimization Motivating Example Applications Least-squares(LS) and

Introduction: Mathematical optimization Motivating Example Applications Least-squares(LS) and linear programming(LP) - Very common place Course goals and topics Nonlinear optimization Brief history of convex optimization Prof. G a n e s h

489 views β€’ 31 slides
20.1 Combinatorial Optimization next chapters: combinatorial optimization similar scenario,

20.1 Combinatorial Optimization next chapters: combinatorial optimization similar scenario,

Foundations of Artificial Intelligence April 1, 2019 20. Combinatorial Optimization: Introduction and Hill-Climbing Foundations of Artificial Intelligence 20.1 Combinatorial Optimization 20. Combinatorial Optimization: Introduction and

491 views β€’ 7 slides
UNIT 1 An introduction to optimization An optimization problem An engineering factory makes 4

UNIT 1 An introduction to optimization An optimization problem An engineering factory makes 4

UNIT 1 An introduction to optimization An optimization problem An engineering factory makes 4 products (PROD1 to PROD4) on the following machines: 4 grinders, 2 drills and 1 planer. Each product yields a certain contribution to profit (defined

447 views β€’ 30 slides
Convex Optimization by Stephen Boyd, and Lieven Vandenberghe. Optimization for Machine Learning by

Convex Optimization by Stephen Boyd, and Lieven Vandenberghe. Optimization for Machine Learning by

Convex Optimization by Stephen Boyd, and Lieven Vandenberghe. Optimization for Machine Learning by Suvrit Sra, Sebastian Nowozin, and Stephen J. Wright Convex Optimization- Chapter 1: Introduction mathematical optimization Least-squares and

764 views β€’ 20 slides
AM 205: lecture 17 Last time: introduction to optimization Today: scalar and vector

AM 205: lecture 17 Last time: introduction to optimization Today: scalar and vector

AM 205: lecture 17 Last time: introduction to optimization Today: scalar and vector optimization Note: last years midterm is now available on the website for practice Motivation: Optimization If the objective function or any of the

1.1k views β€’ 48 slides
P2P Combinatorial Optimization Amir H. Payberah (amir@sics.se) P2P Combinatorial Optimization, 13

P2P Combinatorial Optimization Amir H. Payberah (amir@sics.se) P2P Combinatorial Optimization, 13

P2P Combinatorial Optimization Amir H. Payberah (amir@sics.se) P2P Combinatorial Optimization, 13 th October 2009 1 Agenda Introduction to Optimization Metaheuristics in Combinatorial Optimization P2P Combinatorial Optimization P2P

583 views β€’ 45 slides
MATH-529 Fundamentals of Optimization Introduction Marco A. Montes de Oca Mathematical

MATH-529 Fundamentals of Optimization Introduction Marco A. Montes de Oca Mathematical

MATH-529 Fundamentals of Optimization Introduction Marco A. Montes de Oca Mathematical Sciences, University of Delaware, USA 1 / 18 What is optimization? Definition Optimization is the process of finding the best element of a set. Some

863 views β€’ 18 slides
Optimization++ Complexities and strategies of optimization Instruction Scheduling

Optimization++ Complexities and strategies of optimization Instruction Scheduling

Optimization++ Complexities and strategies of optimization Instruction Scheduling Register Allocation Optimization Recap 1. Intermediate language (IL) module - better separation of front-end and back-end modules - permit

583 views β€’ 16 slides
AM 205: lecture 20 Today: PDE optimization, constrained optimization example New topic:

AM 205: lecture 20 Today: PDE optimization, constrained optimization example New topic:

AM 205: lecture 20 Today: PDE optimization, constrained optimization example New topic: eigenvalue problems PDE Constrained Optimization We will now consider optimization based on a function that depends on the solution of a PDE Let us

604 views β€’ 26 slides
Lecture 19 Additional Material: Optimization CS109A Introduction to Data Science Pavlos

Lecture 19 Additional Material: Optimization CS109A Introduction to Data Science Pavlos

Lecture 19 Additional Material: Optimization CS109A Introduction to Data Science Pavlos Protopapas and Kevin Rader Outline Optimization Challenges in Optimization Momentum Adaptive Learning Rate Parameter Initialization Batch

457 views β€’ 45 slides
Introduction to Convex Optimization Xuezhi Wang Computer Science Department Carnegie Mellon

Introduction to Convex Optimization Xuezhi Wang Computer Science Department Carnegie Mellon

Convexity Unconstrained Convex Optimization Constrained Optimization Introduction to Convex Optimization Xuezhi Wang Computer Science Department Carnegie Mellon University 10701-recitation, Jan 29 Introduction to Convex Optimization

439 views β€’ 22 slides
solving Linear Optimization What we did so far How to model an optimization problem

solving Linear Optimization What we did so far How to model an optimization problem

Chapter 1 Linear Programming Paragraph 2 Developing Ideas for an Algorithm solving Linear Optimization What we did so far How to model an optimization problem Data, Variables, Constraints, Objective We have seen that optimization

483 views β€’ 22 slides
1. Introduction Logistics Optimization examples About the course Detailed example

1. Introduction Logistics Optimization examples About the course Detailed example

CS/ECE/ISyE 524 Introduction to Optimization Spring 201718 1. Introduction Logistics Optimization examples About the course Detailed example with code Assignment #0 Laurent Lessard (www.laurentlessard.com) Logistics

378 views β€’ 25 slides
MATH529 Fundamentals of Optimization Unconstrained Optimization II Marco A. Montes de Oca

MATH529 Fundamentals of Optimization Unconstrained Optimization II Marco A. Montes de Oca

MATH529 Fundamentals of Optimization Unconstrained Optimization II Marco A. Montes de Oca Mathematical Sciences, University of Delaware, USA 1 / 31 Recap 2 / 31 Optimization via Calculus Example Find the local and global minimizers and

2.41k views β€’ 26 slides
CS675: Convex and Combinatorial Optimization Spring 2018 Convex Optimization Problems

CS675: Convex and Combinatorial Optimization Spring 2018 Convex Optimization Problems

CS675: Convex and Combinatorial Optimization Spring 2018 Convex Optimization Problems Instructor: Shaddin Dughmi Outline Convex Optimization Basics 1 Common Classes 2 Interlude: Positive Semi-Definite Matrices 3 More Convex Optimization

887 views β€’ 61 slides
Search Engine Optimization What is Search Engine Optimization Search Engine Optimization is the

Search Engine Optimization What is Search Engine Optimization Search Engine Optimization is the

Search Engine Optimization What is Search Engine Optimization Search Engine Optimization is the process of improving the visibility of a website on organic (&quot;natural&quot; or un-paid) search engine result pages (SERPs), by incorporating

1.54k views β€’ 35 slides
Optimization Unconstrained optimization Constrained optimization Newton with equality

Optimization Unconstrained optimization Constrained optimization Newton with equality

Optimization Unconstrained optimization Constrained optimization Newton with equality One-dimensional constraints Multi-dimensional Active-set method Newtons method Descent methods Basic Newton Gradient Simplex method Gauss-

795 views β€’ 55 slides
Convex Optimization 4. Convex Optimization Problems Prof. Ying Cui Department of Electrical

Convex Optimization 4. Convex Optimization Problems Prof. Ying Cui Department of Electrical

Convex Optimization 4. Convex Optimization Problems Prof. Ying Cui Department of Electrical Engineering Shanghai Jiao Tong University 2018 SJTU Ying Cui 1 / 64 Outline Optimization problems Convex optimization Linear optimization

816 views β€’ 64 slides
PYTHON FOR OPTIMIZATION BEN MORAN @BENM HTTP://BENMORAN.WORDPRESS.COM/ 2014-02-22

PYTHON FOR OPTIMIZATION BEN MORAN @BENM HTTP://BENMORAN.WORDPRESS.COM/ 2014-02-22

PYTHON FOR OPTIMIZATION BEN MORAN @BENM HTTP://BENMORAN.WORDPRESS.COM/ 2014-02-22 INTRODUCTION Python for optimization Not optimizing Python programs Not website optimization/SEO Mathematical optimization! scipy.optimize and friends

326 views β€’ 32 slides
Introduction Query Execution Engine Implements a set of physical operators 2 key

Introduction Query Execution Engine Implements a set of physical operators 2 key

Overview of Query Optimization in Relational Systems An overview of current query optimization techniques Overview of Query Optimization Provides fundamentals of query in Relational Systems optimization Presenter: Albert Wong

294 views β€’ 4 slides

More recommend