Computational Optimization Practical Linesearch 2/12 Line Search - PowerPoint PPT Presentation

Computational Optimization Practical Linesearch 2/12

Line Search type Algorithm Start with x 0 For k =1,…,K � If x k is optimal then stop � Pick descent direction p k � Perform linesearch to determine x k+1 =x k + α k p k

Outline Linesearch Conditions � Armijo + backtracking search � Wolfe Conditions � Strong Wolfe Conditions � Goldstein (not covered in class) Linesearch Algorithms

Why linesearch? Decide how far to step along a descent direction. Just decreasing is not enough Satisfy certain properties to guarentee global convergence of an algorithm Exact properties depends on the algorithm being used

Armijo Condition + α ≤ + α ∇ ( ) ( ) ( ) ' f x p f x c f x p 1 k k k k k α = + α ( ) ( ) g f x p k k = ∇ '(0 ) ( ) ' g f x p 1 k k 1/2 α g(0)+ c 1 μ g’( α ) g( α ) g(0)+ α g’( α )

Backtracking Search Key point: Stepsize cannot be allowed to go to 0 unless gradient is going to 0. Must have sufficient decrease. Fix δ >0 β∈ (0,1) μ ∈ (0,1) α = δ β β β 0 1 2 max( , , ,...) k which satisfies + α ≤ + μα ∇ ( ) ( ) ( ) ' f x p f x f x p k k k k k k k

Backtracking Search Step is fixed value or small enough to satisfy Armijo condition and not too small Not too small guarentees sufficient decrease for convergence. Easy to implement Initial step size may be too small Good for Newton’s method. Not so hot for steepest descent

Step-Length Selection Approximately solve α = + α ∇ < ( ) ( ) ( ) ' 0 g f x p fo r f x p k k k k efficiently to satisfy some condition Two phases � Bracketing – find interval � Zoom – find step in interval

Quadratic Interpolation Approximate g( α ) with h(0) h(0)=g(0), h’(0) =g’(0), g( α 0 ) α g( α 0 ) α 1 ⎛ ⎞ α − − α α ( ) ( 0 ) ' ( ) g g g ⎜ ⎟ α = α + α + 2 ( ) 0 0 ' ( 0 ) ( 0 ) h g g ⎜ ⎟ α 2 ⎝ ⎠ 0

Quadratic Interpolation α α 2 '( ) Potential step g α = 0 1 α − − α α 2( ( ) (0) '( )) g g g 0 0 α g( α 0 ) α 1

Cubic Interpolation α = α + α + α + 3 2 ( ) '(0) (0) h a b g g ⎛ ⎞⎛ ⎞ ⎡ ⎤ = α − α α − − α 2 2 ( ) (0) '(0) 1 a g g g 0 1 1 1 ⎜ ⎟⎜ ⎟ ⎢ ⎥ α α α − α − α α α − − α 2 2 3 3 ( ) ( ) (0) '(0) ⎣ ⎦ b ⎝ ⎠⎝ ⎠ g g g 0 1 1 0 0 1 0 0 − + − 2 3 '(0) b b ag α = 2 3 a

Initial Step Length Assume this step will be on the order of last step α ∇ = α ∇ ' ' f p f p − − − 1 1 1 o k k k k k or α 0 ∇ ' f p α = α − − 1 1 k k − 1 ∇ o k ' f p Interpolate k k ∇ α , , ' and take as min f f f p − 1 k k k k o − 2( ) f f α = − 1 k k o '(0) g Sometimes min of α min(1.01 ,1) o

Final Step Length Algorithms Read page 60 Chapter 3 Adjust interval length such that interval always contains point satisfying strong Wolfe conditions Zoom in using interpolation to find point satisfying wolfe conditions NOTE – details important and not easy. NW recommend using public domain implementation