Optimization CS3220 - Summer 2008 Jonathan Kaldor Problem Setup - PowerPoint PPT Presentation

Optimization CS3220 - Summer 2008 Jonathan Kaldor

Problem Setup • Suppose we have a function f(x) in one variable (for the moment) • We want to find x’ such that f(x’) is a minimum of the function f(x) • Can have local minimum and global minimum - one is a lot easier to find than the other, though, without special knowledge about the problem

Problem Setup • Can find global minimum for certain types of problems • Our focus: general (possibly nonlinear) functions, and local minima • Finding x’ such that f(x’) is minimal over some local region around x’.

Constrained Versus Unconstrained • Most general problem statement: find x ’ such that f( x ’) is minimized, subject to constraints g ( x ’) ≤ 0 • Function g ( x ) represents constraints on the solution (can be equality or inequality constraints) • Our focus: unconstrained optimization (so no g () function)

Applications • Many applications in sciences • Energy Minimization • Function representing the energy in a system • Find the minimum energy - stable state of system

Applications • Minimum Surface Area • Given some shape, find the minimum surface that matches at the boundaries • Real life example: take some shape and dip it into bubble solution. Bubble solution takes the shape of the minimum surface

Applications • Planning and Scheduling • Schedules for sports teams • Finals schedules • Typically solving constrained optimization problems (we’ll look at unconstrained problems only)

Applications • Protein Folding • Image Recognition • ... and many, many more

Bisection Method • Suppose we have an interval [a,b] and we would like to find a local minimum in that interval. • Evaluate at c = (a+b)/2. Do we gain anything • Answer: no (don’t have any idea which interval the minimum is in

Bisection Method a c b

Trisection Method • If we divide up our interval into three pieces, though, we can tell which interval has a minimum • Suppose we have a < u < v < b. Then if f(u) < f(v), we know that there is a local minimum in the interval [a,v] • Similar case for f(v) < f(u), local minimum in [u,b]

Trisection Method • So we evaluate at u = a + (b-a)/3 and v = a + 2*(b-a)/3. Suppose f(u) < f(v), and our new interval is [a,v] • The u we computed before is the midpoint between a and v. We can’t reuse the function value on the next step, so we need to evaluate our function twice at every iteration

Trisection Method • Note: we aren’t restricted to choosing u and v evenly spaced between a and b • Can we choose u and v such that we can reuse the computation of whichever one doesn’t become an endpoint at the next step?

Trisection Method • More accurately, suppose we are at [a 0 , b 0 ], and we compute u 0 , v 0 . Our next interval happens to be [a 0 , v 0 ], and we now need to compute u 1 , v 1 . Can we arrange our u’s and v’s such that u 0 = u 1 or u 0 = v 1 ? • Note: this will allow us to save a function evaluation at each iteration

Golden Section Search • Define h = ρ (b - a). This is the distance of our u and v from the endpoints a and b. Thus, u = a + h, v = b - h. • We want to find ρ so that u 0 is in the correct position for the next step.

Golden Section Search ρ 1- ρ 0 u 0 v 0 1 1- ρ 0 u 1 v 1

Golden Section Search ρ 1- ρ 0 u 0 v 0 1 ρ 1- ρ = 1- ρ 1 1- ρ 0 u 1 v 1

Golden Section Search • Given the relation between the ratios, we can then compute the desired value of ρ : ρ = 1 - 2 ρ + ρ 2 ρ 2 - 3 ρ + 1 = 0 ρ = (3 - √ 5)/2 = 2 - ϕ ≈ 0.382 where ϕ is the golden ratio (again)

Convergence • At every iteration, we compute a new interval 61.8% the size of the old one • Larger than the interval we compute during bisection, but smaller than the interval we compute using trisection • End up needing around 75 iterations to reduce error in our bounds to epsilon

Existence / Uniqueness • If our function is unimodular on the original interval [a,b] (has only one minimum) then we will converge to it. • If the function has multiple local minima in the interval, we will converge to one of them (but it may not be the global minimum)

Sidestep: Calculus (again) • Suppose we have some function f(x), and we want to find the critical points of f • Minima, maxima, and points of inflection • Critical points are where f’(x) = 0 • Minima: f’(x) = 0, f’’(x) > 0 Maxima: f’(x) = 0, f’’(x) < 0 Point of inflection: f’(x) = f’’(x) = 0

Sidestep: Calculus (again) • So, we want to find roots of f’(x) • We can use the root finding strategies we already talked about! • In particular, we can use Newton’s Method applied to f’(x). Our linear approximation is then f’(x+h) = f’(x 0 ) + f’’(x 0 ) h • So h = -f’(x 0 ) / f’’(x 0 )

Newton’s Method • Another way of deriving it: suppose we are at a point x0. When we wanted to find the root, we took a linear approximation to the function at that point, since it was easy to find the root • What function is easy to find a minimum of? A quadratic function! So take the quadratic approximation of f at x0

Newton’s Method • f(x+h) = f(x 0 ) + f’(x 0 ) h + f’’(x 0 ) h 2 /2 • We can compute the minimum of this function by taking the derivative with respect to h and solving. • End up with h = - f’(x 0 ) / f’’(x 0 ) • Note: same answer either derivation

Newton’s Method • We gain all of the benefits (and drawbacks) of Newton’s Method • In particular, we have no guarantee of convergence if we don’t start reasonably close to the minimum • We have the added wrinkle that we may not even converge to a minimum (could converge to maximum or inflection pt)