Systems of Nonlinear Equations CS3220 - Summer 2008 Jonathan - PowerPoint PPT Presentation

Systems of Nonlinear Equations CS3220 - Summer 2008 Jonathan Kaldor

From 1 to N • So far, we have considered finding roots of scalar equations f(x) = 0 • f(x) takes a single scalar argument, computes a single scalar • Much like for linear systems, we would like to extend this to systems of n equations in n variables

From 1 to N • We can write this as f 1 (x 1 , x 2 , ... x n ) = 0 f 2 (x 1 , x 2 , ... x n ) = 0 ... f n (x 1 , x 2 , ... x n ) = 0 • Or, better yet, f ( x ) = 0 • f ( x ) takes a vector of arguments, produces a vector

Applications • Physical Simulation • Planetary Dynamics • Structural Analysis • Differential Equations

From 1 to N • Just like removing the assumption of linearity complicated things, so to does the addition of multiple equations in multiple variables • In particular, we lose the ability to easily bracket possible roots

Bracketing Roots (or not) • Consider 2 variable, 2 equation case. What does it mean to bracket a root? How can we refine our intervals based on evaluating the function at grid points? • (chalkboard)

Bracketing Roots (or not) • What this means is that we’ve lost our easy and guaranteed convergence method • That’s not to say that there are no methods that guarantee global convergence, though • Beyond scope of this course • In general, more complicated than the simple bisection we studied

Review of Vector Calculus • Let f ( x ) be a function from R n → R n • Define ∂ f i / ∂ x j as the partial derivative of the i-th function with respect to the j-th variable • Derivative of the function with respect to the j-th variable while holding all other variables constant

Review of Vector Calculus • Let g(x, y) = x 2 y + x sin(y) • Then ∂ g/ ∂ x = 2xy + sin(y) ∂ g/ ∂ y = x 2 + x cos(y)

Review of Vector Calculus • We can then define the jacobian (first derivative) of the function by the matrix J ij = ∂ f i / ∂ x j • Note: n x n matrix when f ( x ) is system of n equations in n variables • Also: Jacobian is a function of x : J f ( x )

Review of Vector Calculus • Compute the Jacobian of f 1 (x, y, z) = x 2 + y 2 + z 2 - 10 f 2 (x, y, z) = x sin(z) + y cos(z) f 3 (x, y, z) = e z

Newton’s Method Redux • Newton’s Method is one of the single variable methods that has an extension to n variables and n equations. • Relies on linear approximation of our function • Recall for g(x), we can use the Taylor Series to approximate the function as: g(x 0 +h) = g(x 0 ) + hg’(x 0 ) + h 2 g’’(x 0 ) + ...

Newton’s Method Redux • The Taylor series has an extension to n equations in n variables: f( x + s ) = f( x ) + J f ( x ) s + ... • Quickly gets complicated... • ... but fortunately, we only need the linear approximation

Newton’s Method Redux • Start at a point x 0 • Take the linear approximation of f () at x 0 : f ( x + s ) ≈ f ( x 0 ) + J f ( x 0 ) s • Solve for root of linear approximation: J f ( x 0 ) s = - f ( x 0 ) • Note: n x n analog of division is matrix inversion

Newton’s Method Redux • In order to find our next iterate, we need to solve the linear system J f ( x 0 ) s = - f ( x 0 ) • Newton in one variable replaces a nonlinear equation with a linear equation. In many variables, this simply replaces a system of nonlinear equations with a system of linear equations. • This is often referred to as “linearizing” the system

Newton’s Method Redux • Of course, this is most likely not our root, so we need to iterate: x 1 = initial guess for k = 1, 2, ... Solve J f ( x k ) s k = - f ( x k ) for s k x k+1 = x k + s k end

Newton’s Method Redux • Note: this is why we limit ourselves to n equations in n variables, so that we can compute the inverse of the Jacobian • This also depends on our Jacobian being invertible at each of our (n-dimensional analogue of our derivative being non-zero) • Don’t need to actually compute inverse of Jacobian (of course)

Newton’s Method Redux • Example of Newton’s Method

Newton’s Method Redux • Convergence is quadratic like the “other” Newton’s method, under similar assumptions • Namely, our initial guess should be “relatively close” to the root • Note that as n increases, harder and harder to “guess” a good initial point without additional information about prob.

Newton’s Method Redux • Newton’s Method is quite expensive per iteration: • Each iteration involves computing an n x n Jacobian matrix • And then solving a linear system with that matrix: O(n 3 ) flops

Newton’s Method Redux • As a result, sometimes common to do a limited number of Newton steps and tolerate the error • Oftentimes, only one iteration

Newton’s Method Redux • Minimizing the amount of work per iteration has led to alternate methods, called quasi- Newton methods • Simplest example: compute Jacobian matrix at x k , use it for more than one iteration without recomputing • Slows convergence, since we are not always using the correct derivative

Slightly Safer Strategies • We can also try and improve the quality of our steps by using damped Newton steps: x k+1 = x k + α k s k • Choose α k intelligently so that we dont take any large or dangerous steps (usually by choosing α k < 1). As we get close to the answer, choose larger α k for quick convergence

Slightly Safer Strategies • Note: this still relies on picking a good α k value • We can try and choose α k so that ‖ f ( x k+1 ) ‖ < ‖ f ( x k ) ‖ • We are slowly creeping towards the answer • Still no absolute guarantees it will converge