2 Numerical Mathematics — Iteration schemes 2.1 The Newton-Raphson method This is the usual workhorse not only for solving for the roots of a single equation but also for finding the roots of coupled system of nonlinear algebraic equations. In the present notes we shall cover only one equation in one unknown. I will derive the formula in two different ways: (i) graphical; (ii) mathematical via Taylor’s series.
2.1.1 Graphical Motivation • y = f ( x ) f ( x 0 ) = f ′ ( x 0 ) δ f ( x 0 ) = ⇒ = δ f ′ ( x 0 ) f ( x 0 ) f ( x 0 ) = ⇒ x 0 − x 1 = f ′ ( x 0 ) x 0 − f ( x 0 ) = ⇒ x 1 = f ′ ( x 0 ) . • x x 1 x 0 x 2 δ The tangent to y = f ( x ) at x = x 0 is drawn, and the next iterate is defined to be the point where the tangent crosses the horizontal axis. In general: x n +1 = x n − f ( x n ) The Newton-Raphson method f ′ ( x n ) .
2.1.2 Mathematical Motivation Let x be the exact solution, x n , the current iterate and ǫ its error. x = x n + ǫ (1) Hence 0 = f ( x ) by definition = f ( x n + ǫ ) also by definition (2) = f ( x n ) + ǫf ′ ( x n ) + · · · two terms of the Taylor’s series. ǫ ≃ − f ( x n ) = ⇒ f ′ ( x n ) , Given that this is an approximation for ǫ , we need to modify Eq. (1) to the form, x n − f ( x n ) = x n + ǫ = x n +1 f ′ ( x n ) .
2.2 An example of the use of the Newton-Raphson method We will look at the main example used for the ad hoc methods, namely to find the roots of x 3 − 7 x +6 = 0 . Using Eq. (2) the formula for the Newton-Raphson method is x n − x 3 2 x 3 n − 7 x n + 6 n − 6 x n +1 = = n − 7 . (3) 3 x 2 n − 7 3 x 2 Three cases: x 0 = 1 . 1 x 1 = 0 . 990 504 451 038 5755 = 0 . 999 933 743 233 6637 x 2 = 0 . 999 999 996 708 0032 x 3 x 4 = 0 . 999 999 999 999 9999 = 2 . 1 x 0 = 2 . 009 951 845 906 9023 x 1 = 2 . 000 116 453 000 0373 x 2 x 3 = 2 . 000 000 016 269 6456 = 2 . 000 000 000 000 0004 x 4 = − 2 . 9 x 0 = − 3 . 004 827 207 899 0674 x 1 x 2 = − 3 . 000 010 451 675 8714 = − 3 . 000 000 000 049 1567 x 3 = − 3 . 000 000 000 000 0000 x 4
There are various things to notice here. The first is that all three roots have been obtained using the method. The second is the astonishing speed of convergence, especially when compared with the somewhat weedy performance of the ad hoc methods. When the error in an iterate is small, the error in the next iterate is the square of that, rather than a constant multiplied by it. I have displayed 16 decimal places and coloured the correct decimal places in red in the above computations to enable the speed of convergence to be shown clearly. We may use a perturbation analysis to model this.
2.3 Perturbation analysis x n − x 3 2 x 3 n − 7 x n + 6 n − 6 x n +1 = = n − 7 . (3) 3 x 2 n − 7 3 x 2 In the first instance let us consider the root, x = 1 , by setting x n = 1 + ǫ as before and where | ǫ | ≪ 1 . Using the red expression above. 2(1 + ǫ ) 3 − 6 = using (3) x n +1 3(1 + ǫ ) 2 − 7 2 + 6 ǫ + 6 ǫ 2 + 2 ǫ 3 − 6 = − 4 + 6 ǫ + 6 ǫ 2 + 2 ǫ 3 = tidying up 3 + 6 ǫ + 3 ǫ 2 − 7 − 4 + 6 ǫ + 3 ǫ 2 − 4 + 6 ǫ + (3 ǫ 2 + 3 ǫ 2 ) + 2 ǫ 3 = to reproduce the denominator − 4 + 6 ǫ + 3 ǫ 2 3 ǫ 2 + 2 ǫ 3 = 1 + tidying up − 4 + 6 ǫ + 3 ǫ 2 4 ǫ 2 + · · · 1 − 3 = using leading order terms. It is possible to get to this point by using the blue formula in Eq. (3), but the analysis takes much longer. Conclusion: the error in x n +1 is proportional to the square of the error in x n . The Newton-Raphson method converges quadratically.
For the root, x = 2 we’ll follow the same sort of analysis using x n = 2 + ǫ to start with. Hence 2(2 + ǫ ) 3 − 6 = using (3) x n +1 3(2 + ǫ ) 2 − 7 2(8 + 12 ǫ + 6 ǫ 2 + ǫ 3 ) − 6 = 3(4 + 4 ǫ + ǫ 2 ) − 7 10 + 24 ǫ + 12 ǫ 2 + 2 ǫ 3 = 5 + 12 ǫ + 3 ǫ 2 10 + 24 ǫ + (6 ǫ 2 + 6 ǫ 2 ) + 2 ǫ 3 = looking for 2 × the denominator 5 + 12 ǫ + 3 ǫ 2 2(5 + 12 ǫ + 3 ǫ 2 ) + 6 ǫ 2 + 2 ǫ 3 = 5 + 12 ǫ + 3 ǫ 2 6 ǫ 2 + 2 ǫ 3 = 2 + 5 + 12 ǫ + 3 ǫ 2 5 ǫ 2 + · · · 2 + 6 = using leading order terms. Again we have quadratic convergence, but don’t be fooled that the coefficient of ǫ 2 is multiplied by a constant which is greater than 1 — this is still quadratic convergence
We shall also look at the third root, x = − 3 , for the sake of completeness, but setting x n = − 3 + ǫ . Hence, 2( − 3 + ǫ ) 3 − 6 = using (3) x n +1 3( − 3 + ǫ ) 2 − 7 2( − 27 + 27 ǫ − 9 ǫ 2 + ǫ 3 ) − 6 = 3(9 − 6 ǫ + ǫ 2 ) − 7 − 60 + 54 ǫ − 18 ǫ 2 + 2 ǫ 3 = 20 − 18 ǫ + 3 ǫ 2 − 3(20 − 18 ǫ + 3 ǫ 2 ) − 9 ǫ 2 + 2 ǫ 3 = 20 − 18 ǫ + 3 ǫ 2 − 9 ǫ 2 + 2 ǫ 3 = − 3 + 20 − 18 ǫ + 3 ǫ 2 20 ǫ 2 + · · · − 3 − 9 = using leading order terms. For a third time we obtain quadratic convergence.
Note: In summary, we may say the following: • If we have x n = x exact + ǫ , then x n +1 = x exact + cǫ 2 + · · · . • When the initial iterate is close enough to the exact solution, then the Newton-Raphson method will converge quadratically to that root. The value of c here is irrelevant because it multiplies ǫ 2 . • The Newton-Raphson method performs extremely well when the roots are single roots. The next task is to find out how well it performs for double roots.
2.4 Application of the Newton-Raphson method to a double root. Let us consider how to find the roots of x 3 − 12 x + 16 = 0 . This cubic was created from ( x − 2) 2 ( x + 4) , so the roots are 2 , 2 and − 4 . The Newton-Raphson scheme is, x n − x 3 n − 12 x n + 16 = x n +1 3 x 2 n − 12 2 x 3 n − 16 (4) = 3 x 2 n − 12 2( x 3 n − 8) = n − 4) . 3( x 2 The eagle-eyed will notice that the final fraction has ( x − 2) as a common factor.....
We have: = − 4 . 1 x 0 x 1 = − 4 . 003 174 603 x 2 = − 4 . 000 003 354 Quadratic convergence = − 4 . 000 000 000 Single root x 3 x 0 = 2 . 1 = 2 . 050 407 x 1 = 2 . 025 308 x 2 = 2 . 012 680 Linear convergence x 3 x 4 = 2 . 006 347 Double root We retain quadratic convergence for x = − 4 , the single root. The Newton-Raphson method converges linearly for the double root. Whilst this is slower than for a single root, it is considerably better than for a double root being found with an ad hoc method.
2.5 Perturbation analysis for a double root. Let us consider what happens to small errors for this double root. We shall let x n = 2 + ǫ in Eq. (4), and hence, 3 × (2 + ǫ ) 3 − 8 2 x n +1 = using (4) (2 + ǫ ) 2 − 4 3 × (8 + 12 ǫ + 6 ǫ 2 + ǫ 3 − 8) 2 = (4 + 4 ǫ + ǫ 2 − 4) 3 × (12 ǫ + 6 ǫ 2 + ǫ 3 ) 2 = (4 ǫ + ǫ 2 ) 2 + ǫ + 1 6 ǫ 2 = cancelling the common factor, ǫ 1 + 1 4 ǫ Can neglect ǫ 2 terms (2 + ǫ + · · · )(1 − 1 = 4 ǫ + · · · ) and have used the Binomial expansion 2 + 1 = 2 ǫ + · · · For a double root the errors halve every iteration. This is always true for a double root.
2.6 Brief Summary of the Newton-Raphson method. • The Newton-Raphson method is used ubiquitously primarily because it is a nice compromise between the rate of convergence and the complexity of the formula. Ad hoc methods are easier, but they converge much more slowly. • For single roots the Newton-Raphson method displays quadratic convergence for all roots. • For double roots convergence is linear. • Although it hasn’t been demonstrated here, the Newton-Raphson method also converges linearly (though a little slower) for triple roots and so on (see the problem sheet). • Just to note that there are faster methods than the Newton-Raphson method such as Halley’s method, which has cubic convergence. The formulae are much more complicated. • The Newton-Raphson method may be extended to solve N equations in N unknowns.
Recommend
More recommend
