MATH529 Fundamentals of Optimization Unconstrained Optimization II - PowerPoint PPT Presentation

MATH529 – Fundamentals of Optimization Unconstrained Optimization II Marco A. Montes de Oca Mathematical Sciences, University of Delaware, USA 1 / 31

Recap 2 / 31

Optimization via Calculus Example Find the local and global minimizers and maximizers on R of f ( x ) = 3 x 4 − 4 x 3 + 1. 3 / 31

Optimization via Calculus Graph of f ( x ) = 3 x 4 − 4 x 3 + 1. 4 / 31

Optimization via Calculus Two theorems summarize the basic facts about global optimization of one variable functions. Theorem (1st order condition (Necessary, but not sufficient )) Suppose that f ( x ) is a differentiable function on R (or the function’s domain I). If x ⋆ is a global minimizer of f ( x ) , then f ′ ( x ⋆ ) = 0 . 5 / 31

Optimization via Calculus Two theorems summarize the basic facts about global optimization of one variable functions. Theorem (1st order condition (Necessary, but not sufficient )) Suppose that f ( x ) is a differentiable function on R (or the function’s domain I). If x ⋆ is a global minimizer of f ( x ) , then f ′ ( x ⋆ ) = 0 . Theorem (2nd order condition (Sufficient, but not necessary )) Suppose that f ( x ) , f ′ ( x ) , and f ′′ ( x ) are all continuous on R (or I) and that x ⋆ is a critical point of f ( x ) . a) If f ′′ ( x ) ≥ 0 for all x ∈ R (or I), then x ⋆ is a global minimizer of f ( x ) on R (or I). b) If f ′′ ( x ) > 0 for all x ∈ I such that x � = x ⋆ , then x ⋆ is a strict global minimizer of f ( x ) on R (or I). 6 / 31

Optimization via Calculus Local optimization is easier to verify. Theorem (1st order condition (Necessary, but not sufficient )) Suppose that f ( x ) is a differentiable function on R (or I). If x ⋆ is a local minimizer of f ( x ) , then f ′ ( x ⋆ ) = 0 . 7 / 31

Optimization via Calculus Local optimization is easier to verify. Theorem (1st order condition (Necessary, but not sufficient )) Suppose that f ( x ) is a differentiable function on R (or I). If x ⋆ is a local minimizer of f ( x ) , then f ′ ( x ⋆ ) = 0 . Theorem (2nd order condition (Sufficient, but not necessary )) Suppose that f ( x ) , f ′ ( x ) , and f ′′ ( x ) are all continuous on R (or I) and that x ⋆ is a critical point of f ( x ) . If f ′′ ( x ⋆ ) > 0 , then x ⋆ is a strict local minimizer of f ( x ) . 8 / 31

Optimization via Calculus Exercise Find the local and global minimizers and maximizers on I = ( − 1 , 1) of f ( x ) = ln(1 − x 2 ). 9 / 31

Optimization via Calculus What about functions of many variables? 10 / 31

Optimization via Calculus What about functions of many variables? Extend theorems that allow us to identify and classifly local minimizers of one variable functions to multivariable cases. 11 / 31

Optimization via Calculus Notation:   x 1 x 2     A vector in R n is an ordered n -tuple x = x 3 of real numbers    .  .   .   x n called components of x . If x and y are vectors in R n , then their dot product or inner product is defined by   y 1 y 2   n   x · y = x T y = ( x 1 , x 2 , x 3 , . . . , x n ) y 3 � = x i y i    .  .   i =1 .   y n where x T is the transpose of x . 12 / 31

Optimization via Calculus Notation: If f ( x ) is a function of n variables with continuous first and second partial derivatives on R n , then the gradient of f ( x ) is the vector ∂ f   ∂ x 1 ∂ f   ∂ x 2     ∂ f   ∇ f = ∂ x 3     . .   .     ∂ f ∂ x n 13 / 31

Optimization via Calculus Notation: The Hessian of f ( x ), denoted by ∇ 2 f or Hf , is the symmetric n × n matrix ∂ 2 f ∂ 2 f ∂ 2 f ∂ 2 f   . . . ∂ x 2 ∂ x 1 ∂ x 2 ∂ x 1 ∂ x 3 ∂ x 1 ∂ x n 1   ∂ 2 f ∂ 2 f ∂ 2 f ∂ 2 f  . . .   ∂ x 2  ∂ x 2 ∂ x 1 ∂ x 2 ∂ x 3 ∂ x 2 ∂ x n 2     ∂ 2 f ∂ 2 f ∂ 2 f ∂ 2 f ∇ 2 f = Hf = . . .   ∂ x 3 ∂ x 1 ∂ x 3 ∂ x 2 ∂ x 2 ∂ x 3 ∂ x n   3     . . . . ... . . . .   . . . .       ∂ 2 f ∂ 2 f ∂ 2 f ∂ 2 f . . . ∂ x 2 ∂ x n ∂ x 1 ∂ x n ∂ x 2 ∂ x n ∂ x 3 n 14 / 31

Optimization via Calculus Definition Suppose f ( x ) is a real-valued function defined on a subset D of R n . A point x ⋆ in D is: A global minimizer for f ( x ) on D if f ( x ⋆ ) ≤ f ( x ) for all x ∈ D ; 15 / 31

Optimization via Calculus Definition Suppose f ( x ) is a real-valued function defined on a subset D of R n . A point x ⋆ in D is: A global minimizer for f ( x ) on D if f ( x ⋆ ) ≤ f ( x ) for all x ∈ D ; A strict global minimizer for f ( x ) on D if f ( x ⋆ ) < f ( x ) for all x ∈ D such that x � = x ⋆ ; 16 / 31

Optimization via Calculus Definition Suppose f ( x ) is a real-valued function defined on a subset D of R n . A point x ⋆ in D is: A global minimizer for f ( x ) on D if f ( x ⋆ ) ≤ f ( x ) for all x ∈ D ; A strict global minimizer for f ( x ) on D if f ( x ⋆ ) < f ( x ) for all x ∈ D such that x � = x ⋆ ; A local minimizer for f ( x ) if there is a positive number δ such that f ( x ⋆ ) ≤ f ( x ) for all x ∈ D for which x in B ( x ⋆ , δ ); 17 / 31

Optimization via Calculus Definition Suppose f ( x ) is a real-valued function defined on a subset D of R n . A point x ⋆ in D is: A global minimizer for f ( x ) on D if f ( x ⋆ ) ≤ f ( x ) for all x ∈ D ; A strict global minimizer for f ( x ) on D if f ( x ⋆ ) < f ( x ) for all x ∈ D such that x � = x ⋆ ; A local minimizer for f ( x ) if there is a positive number δ such that f ( x ⋆ ) ≤ f ( x ) for all x ∈ D for which x in B ( x ⋆ , δ ); A strict local minimizer for f ( x ) if there is a positive number δ such that f ( x ⋆ ) < f ( x ) for all x ∈ D for which x in B ( x ⋆ , δ ) and x � = x ⋆ ; 18 / 31

Optimization via Calculus Definition Suppose f ( x ) is a real-valued function defined on a subset D of R n . A point x ⋆ in D is: A global minimizer for f ( x ) on D if f ( x ⋆ ) ≤ f ( x ) for all x ∈ D ; A strict global minimizer for f ( x ) on D if f ( x ⋆ ) < f ( x ) for all x ∈ D such that x � = x ⋆ ; A local minimizer for f ( x ) if there is a positive number δ such that f ( x ⋆ ) ≤ f ( x ) for all x ∈ D for which x in B ( x ⋆ , δ ); A strict local minimizer for f ( x ) if there is a positive number δ such that f ( x ⋆ ) < f ( x ) for all x ∈ D for which x in B ( x ⋆ , δ ) and x � = x ⋆ ; A critical point (also called a stationary point ) of f ( x ) if the first partial derivatives of f ( x ) exist at x ⋆ and ∂ f ∂ x i = 0, for i = 1 , 2 , 3 . . . , n . 19 / 31

Optimization via Calculus Theorem (Multivariable Taylor’s formula) Suppose that x , x ⋆ are points in R n and that f ( x ) is a real-valued function of n variables with continuous first and second partial derivatives on some open set containing the line segment [ x ⋆ , x ] = { w ∈ R n : w = x ⋆ + t ( x − x ⋆ ) , 0 ≤ t ≤ 1 } joining x ⋆ and x . Then, there exists a z ∈ [ x ⋆ , x ] such that f ( x ) = f ( x ⋆ ) + ∇ f ( x ⋆ ) T ( x − x ⋆ ) + 1 2( x − x ⋆ ) T Hf ( z )( x − x ⋆ ) 20 / 31

Optimization via Calculus Theorem (Local minimizer identification) Suppose that f ( x ) is a real-valued function for which all first partial derivatives of f ( x ) exist on a subset D ∈ R n . If x ⋆ is an interior point of D that is a local minimizer of f ( x ) , then ∇ f ( x ⋆ ) = 0 . 21 / 31

Optimization via Calculus Theorem (Classification of minimizers (maximizers)) Suppose that x ⋆ is a critical point of a function f ( x ) with continuous first and second partial derivatives on R n . Then: x ⋆ is a global minimizer of f ( x ) if ( x − x ⋆ ) T Hf ( z )( x − x ⋆ ) ≥ 0 for all x ∈ R n and all z ∈ [ x ⋆ , x ] ; x ⋆ is a strict global minimizer of f ( x ) if ( x − x ⋆ ) T Hf ( z )( x − x ⋆ ) > 0 for all x ∈ R n such that x � = x ⋆ and for all z ∈ [ x ⋆ , x ] ; x ⋆ is a global maximizer of f ( x ) if ( x − x ⋆ ) T Hf ( z )( x − x ⋆ ) ≤ 0 for all x ∈ R n and all z ∈ [ x ⋆ , x ] ; x ⋆ is a strict global maximizer of f ( x ) if ( x − x ⋆ ) T Hf ( z )( x − x ⋆ ) < 0 for all x ∈ R n such that x � = x ⋆ and for all z ∈ [ x ⋆ , x ] ; 22 / 31

Optimization via Calculus Practical ways to use the previous theorem: Conditions that involve the form ( x − x ⋆ ) T Hf ( z )( x − x ⋆ ), or in general v T A v , where A is a symmetric square matrix, call for methods to identify whether A (in our case the Hessian of the objective function) is positive or negative (semi)definite . 23 / 31

Optimization via Calculus Quadratic forms: Let   a 11 a 12 . . . a 1 n a 21 a 22 . . . a 2 n   A =  .  . . .  ... . . .   . . .  a n 1 a n 2 . . . a nn The quadratic form Q A ( x ) = x T A x = a 11 x 2 1 + a 12 x 1 x 2 + a 13 x 1 x 3 + . . . + a ij x i x j + . . . + a ii x 2 i + . . . + a nn x 2 n . 24 / 31

Optimization via Calculus Example Write the quadratic form associated with the following matrix:   − 1 0 2 − 3 0 2 1 / 2 − 1   A =  .   2 1 / 2 0 4  − 3 − 1 4 5 25 / 31