Continuous Optimization Sanzheng Qiao Department of Computing and - PowerPoint PPT Presentation

Intro Golden Section Multivariate Functions Least Squares Nonlinear Least Squares Software Continuous Optimization Sanzheng Qiao Department of Computing and Software McMaster University March, 2009

Intro Golden Section Multivariate Functions Least Squares Nonlinear Least Squares Software Outline Introduction 1 Golden Section Search 2 Multivariate Functions 3 Steepest Descent Method Linear Least Squares Problem 4 Nonlinear Least Squares 5 Newton’s Method Gauss-Newton Method Software Packages 6

Intro Golden Section Multivariate Functions Least Squares Nonlinear Least Squares Software Outline Introduction 1 Golden Section Search 2 Multivariate Functions 3 Steepest Descent Method Linear Least Squares Problem 4 Nonlinear Least Squares 5 Newton’s Method Gauss-Newton Method Software Packages 6

Intro Golden Section Multivariate Functions Least Squares Nonlinear Least Squares Software Problem setting Single variable functions. Minimization: x ∈ S f ( x ) min f ( x ) : objective function, single variable and real-valued S : support

Intro Golden Section Multivariate Functions Least Squares Nonlinear Least Squares Software Outline Introduction 1 Golden Section Search 2 Multivariate Functions 3 Steepest Descent Method Linear Least Squares Problem 4 Nonlinear Least Squares 5 Newton’s Method Gauss-Newton Method Software Packages 6

Intro Golden Section Multivariate Functions Least Squares Nonlinear Least Squares Software Golden section search Assumption: f ( x ) has a unique global minimum in [ a , b ] .

Intro Golden Section Multivariate Functions Least Squares Nonlinear Least Squares Software Golden section search Assumption: f ( x ) has a unique global minimum in [ a , b ] . If x ∗ is the minimizer, then f ( x ) monotonically decreases in [ a , x ∗ ] and monotonically increases in [ x ∗ , b ] .

Intro Golden Section Multivariate Functions Least Squares Nonlinear Least Squares Software Golden section search Assumption: f ( x ) has a unique global minimum in [ a , b ] . If x ∗ is the minimizer, then f ( x ) monotonically decreases in [ a , x ∗ ] and monotonically increases in [ x ∗ , b ] . Algorithm Choose interior points c , d : c = a + r ( b − a ) d = a + ( 1 − r )( b − a ) , 0 < r < 0 . 5 if f ( c ) ≤ f ( d ) b = d else a = c end Each step, the length of the interval is reduced by a factor of ( 1 − r ) .

Intro Golden Section Multivariate Functions Least Squares Nonlinear Least Squares Software Golden section search (cont.) The choice of r : When f ( c ) ≤ f ( d ) , d + = c (the next d is c ) When f ( c ) > f ( d ) , c + = d (the next c is d ) Why? Reduce the number of function evaluations

Intro Golden Section Multivariate Functions Least Squares Nonlinear Least Squares Software Choice of r When f ( c ) ≤ f ( d ) , b + = d , d + = a + ( 1 − r )( b + − a ) = a + ( 1 − r )( d − a ) then d + = c means a + ( 1 − r )( d − a ) = a + r ( b − a ) which implies ( 1 − r ) 2 = r . When f ( c ) > f ( d ) , a + = c , then c + = d means c + = c + r ( b − c ) = a + ( 1 − r )( b − a ) which also implies ( 1 − r ) 2 = r . Thus we have √ r = 3 − 5 2

Intro Golden Section Multivariate Functions Least Squares Nonlinear Least Squares Software Algorithm c = a + r*(b - a); fc = f(c); d = a + (1-r)*(b - a); fd = f(d); if fc <= fd b = d; fb = fd; d = c; fd = fc; c = a + r*(b-a); fc = f(c); else a = c; fa = fc; c = d; fc = fd; d = a + (1-r)*(b-a); fd = f(d); end

Intro Golden Section Multivariate Functions Least Squares Nonlinear Least Squares Software Convergence and termination Convergence rate: Each step reduces the length of the interval by a factor of √ 1 − r = 1 − 3 − 5 ≈ 0 . 618 2

Intro Golden Section Multivariate Functions Least Squares Nonlinear Least Squares Software Convergence and termination Convergence rate: Each step reduces the length of the interval by a factor of √ 1 − r = 1 − 3 − 5 ≈ 0 . 618 2 Termination criteria: ( d − c ) ≤ u max ( | c | , | d | ) or a tolerance.

Intro Golden Section Multivariate Functions Least Squares Nonlinear Least Squares Software Outline Introduction 1 Golden Section Search 2 Multivariate Functions 3 Steepest Descent Method Linear Least Squares Problem 4 Nonlinear Least Squares 5 Newton’s Method Gauss-Newton Method Software Packages 6

Intro Golden Section Multivariate Functions Least Squares Nonlinear Least Squares Software Problem setting min f ( x ) where x is a vector (of variables x 1 , x 2 , ..., x n ).

Intro Golden Section Multivariate Functions Least Squares Nonlinear Least Squares Software Problem setting min f ( x ) where x is a vector (of variables x 1 , x 2 , ..., x n ). Gradient   ∂ f ( x c ) ∂ x 1   . ∇ f ( x c ) =   . .   ∂ f ( x c ) ∂ x n

Intro Golden Section Multivariate Functions Least Squares Nonlinear Least Squares Software Problem setting min f ( x ) where x is a vector (of variables x 1 , x 2 , ..., x n ). Gradient   ∂ f ( x c ) ∂ x 1   . ∇ f ( x c ) =   . .   ∂ f ( x c ) ∂ x n −∇ f ( x c ) : the direction of greatest decrease from x c

Intro Golden Section Multivariate Functions Least Squares Nonlinear Least Squares Software Steepest descent method Idea: Steepest descent direction: s c = −∇ f ( x c ) ; Find λ c such that f ( x c + λ c s c ) ≤ f ( x c + λ s c ) , for all λ ∈ R (single variable minimization problem); x + = x c + λ c s c .

Intro Golden Section Multivariate Functions Least Squares Nonlinear Least Squares Software Steepest descent method Idea: Steepest descent direction: s c = −∇ f ( x c ) ; Find λ c such that f ( x c + λ c s c ) ≤ f ( x c + λ s c ) , for all λ ∈ R (single variable minimization problem); x + = x c + λ c s c . Remark. Conjugate gradient method: Use conjugate gradient to replace gradient.

Intro Golden Section Multivariate Functions Least Squares Nonlinear Least Squares Software Outline Introduction 1 Golden Section Search 2 Multivariate Functions 3 Steepest Descent Method Linear Least Squares Problem 4 Nonlinear Least Squares 5 Newton’s Method Gauss-Newton Method Software Packages 6

Intro Golden Section Multivariate Functions Least Squares Nonlinear Least Squares Software Problem setting Given a matrix A ( m -by- n , m ≥ n ) and b ( m -by-1), find x ( n -by-1) minimizing � Ax − b � 2 2 . Example. Square root problem revisited. Find a 1 and a 2 in y ( x ) = a 1 x + a 2 , such that √ √ √ 0 . 25 ) 2 + ( y ( 0 . 5 ) − 0 . 5 ) 2 + ( y ( 1 . 0 ) − ( y ( 0 . 25 ) − 1 . 0 ) 2 is minimized. In matrix-vector form: √     � a 1 � 0 . 25 1 0 . 25 √ A =  , x = , b =    . 0 . 5 1 0 . 5 a 2 √ 1 . 0 1 1 . 0

Intro Golden Section Multivariate Functions Least Squares Nonlinear Least Squares Software Method Transform A into a triangular matrix: � R � PA = 0 where R is upper triangular. Then the problem becomes � Ax − b � 2 2 = � P − 1 ( � Rx − Pb ) � 2 2 � R where � R = � . 0

Intro Golden Section Multivariate Functions Least Squares Nonlinear Least Squares Software Method (cont.) Desirable properties of P : P − 1 is easy to compute; � P − 1 z � 2 2 = � z � 2 2 for any z . Partitioning � b 1 � Pb = , b 2 then the LS solution is the solution of the triangular system Rx = b 1 .

Intro Golden Section Multivariate Functions Least Squares Nonlinear Least Squares Software Choice of P Orthogonal matrix (transformation) Q : Q − 1 = Q T . Example. Givens rotation � � cos θ sin θ G = − sin θ cos θ Introducing a zero into a 2-vector: � x 1 � × � � G = x 2 0 i.e., rotate x onto x 1 -axis.

Intro Golden Section Multivariate Functions Least Squares Nonlinear Least Squares Software Givens rotation x 1 x 2 cos θ = sin θ = � � x 2 1 + x 2 x 2 1 + x 2 2 2 Algorithm. if x(2) = 0 c =1.0; s = 0.0; elseif abs(x(2)) >= abs(x(1)) ct = x(1)/x(2); s = 1/sqrt(1 + ct*ct); c = s*ct; else t = x(2)/x(1); c = 1/sqrt(1 + t*t); s = c*t; end

Intro Golden Section Multivariate Functions Least Squares Nonlinear Least Squares Software Givens rotation (cont.) In general,   c s 0 0   0 1 0 0 G 13 =   − s c   0 0 0 0 0 1     x 1 × x 2 x 2     G 13      = x 3    0 x 4 x 4 Select a pair ( x i , x j ) , find a rotation G ij to eliminate x j .

Intro Golden Section Multivariate Functions Least Squares Nonlinear Least Squares Software QR factorization       × × × × × × × × ×       ⊗ × × 0 × × 0 × ×        − →  − →     × × × ⊗ × × 0 × × × × × × × × ⊗ × ×       × × × × × × × × ×       × × × × × × 0 0 0       − →  − →  − →     0 ⊗ × 0 0 × 0 0 × 0 × × 0 ⊗ × 0 0 ⊗ � R � G 34 G 24 G 23 G 14 G 13 G 12 A = 0 Q = G T 12 G T 13 G T 14 G T 23 G T 24 G T 34 A = QR