Error analysis Definition (Order of convergence) Suppose p n p . If - PowerPoint PPT Presentation

Error analysis Definition (Order of convergence) Suppose p n → p . If ∃ λ, α > 0 s.t. | p n +1 − p | lim | p n − p | α = λ n →∞ then { p n } is said to converge to p of order α , with asymptotic error constant λ . Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 56

Error analysis Definition (Convergence order of numerical methods) An iterative method p n = g ( p n − 1 ) is of order α if the generated { p n } converges to the solution p of p = g ( p ) at order α . In particular: ◮ α = 1: linearly convergent ◮ α = 2: quadratically convergent Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 57

Example Example (Speed comparison: linear vs quadratic) Suppose p n (and q n respectively) converges to 0 linearly (quadratically) with constant 0.5, enumerate the upper bound of | p n | and | q n | . Solution. By definition of convergence order, we know | p n +1 | | q n +1 | lim = 0 . 5 and lim | q n | 2 = 0 . 5 | p n | n →∞ n →∞ Suppose that p 0 and q 0 are close enough to 0 s.t. | p n +1 | / | p n | ≈ 0 . 5 and | q n +1 | / | q n | ≈ 0 . 5 for all n , then | p n | ≈ 0 . 5 | p n − 1 | ≈ 0 . 5 2 | p n − 2 | ≈ · · · ≈ 0 . 5 n | p 0 | | q n | ≈ 0 . 5 | q n − 1 | 2 ≈ 0 . 5 · 0 . 5 2 | q n − 2 | 4 ≈ · · · ≈ 0 . 5 2 n − 1 | q 0 | 2 n Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 58

Example Example (Speed comparison: linear vs quadratic) Suppose p 0 , q 0 ≈ 0 . 5. Then Linear Quadratic 0 . 5 2 n − 1 0 . 5 n n 5 . 0000 × 10 − 1 5 . 0000 × 10 − 1 1 2 . 5000 × 10 − 1 1 . 2500 × 10 − 1 2 1 . 2500 × 10 − 1 7 . 8125 × 10 − 3 3 6 . 2500 × 10 − 2 3 . 0518 × 10 − 5 4 3 . 1250 × 10 − 2 4 . 6566 × 10 − 10 5 1 . 5625 × 10 − 2 1 . 0842 × 10 − 19 6 7 . 8125 × 10 − 3 5 . 8775 × 10 − 39 7 Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 59

Convergence rate of fixed point iteration algorithm Theorem (FPI alg has linear convergence rate) Suppose g ∈ C [ a , b ] s.t. g ( x ) ∈ [ a , b ] , ∀ x ∈ [ a , b ] . If ∃ k ∈ (0 , 1) s.t. | g ′ ( x ) | ≤ k, ∀ x ∈ ( a , b ) , then { p n } generated by FPI algorithm converges to the unique FP of g ( x ) on [ a , b ] linearly . Proof. We already know p n → p where p is the unique fixed point of g by FPI theorem. Also p n +1 − p = g ( p n ) − g ( p ) = g ′ ( ξ ( p n ))( p n − p ) where ξ ( p n ) is between p n and p . So | p n +1 − p | n →∞ | g ′ ( ξ ( p n )) | = | g ′ ( lim n →∞ ξ ( p n )) | = | g ′ ( p ) | ≤ k < 1 lim = lim | p n − p | n →∞ So p n → p linearly with constant k . Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 60

Improve convergence order of FPI to quadratic Theorem (Additional condition for quadratic rate) If g ∈ C 2 [ a , b ] and g ′ ( p ) = 0 for a FP p ∈ ( a , b ) , then ∃ M > 0 s.t. | g ′′ ( x ) | ≤ M, ∀ x ∈ [ a , b ] and ∃ δ > 0 s.t. sequence { p n } by FPI stared in [ p − δ, p + δ ] satisfies | p n +1 − p | ≤ M 2 | p n − p | 2 , ∀ n Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 61

Improve convergence order of FPI Proof. g ∈ C 2 , g ( p ) = p , g ′ ( p ) = 0 together imply ∃ δ > 0 and k ∈ (0 , 1) s.t. | g ′ ( x ) | ≤ k < 1 for all x ∈ [ p − δ, p + δ ] and g : [ p − δ, p + δ ] → [ p − δ, p + δ ]. Also g ( p n ) = g ( p ) + g ′ ( p )( p n − p ) + 1 2 g ′′ ( ξ ( p n ))( p n − p ) 2 where ξ ( p n ) is between p n and p . Since p n +1 = g ( p n ), g ( p ) = p , and g ′ ( p ) = 0, we have p n +1 = p + 1 2 g ′′ ( ξ ( p n ))( p n − p ) 2 . So | p n +1 − p | | p n − p | 2 = 1 2 | g ′′ ( ξ ( p n )) | ≤ M 2 Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 62

Improve convergence order of FPI Suppose we have a fixed point method with g ( x ) = x − φ ( x ) f ( x ). How to choose φ such that FPI converges quadratically? We need g s.t. g ′ ( p ) = 0 at a FP p (root of f ): g ′ ( p ) = 1 − φ ′ ( p ) f ( p ) − φ ( p ) f ′ ( p ) = 0 1 1 Since f ( p ) = 0 we have φ ( p ) = f ′ ( p ) . Choose φ ( x ) = f ′ ( x ) s.t. g ( x ) = x − f ( x ) f ′ ( x ) This is exactly Newton’s method! So Newton’s method converges quadratically. Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 63

Convergence of Newton’s method when f ′ ( p ) = 0 We mentioned condition f ′ ( p ) � = 0 at the root p of f in the convergence proof of Newton’s method above. What if f ′ ( p ) = 0? When will this happen and how to address it? Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 64

Multiple roots f ′ ( p ) = 0 at root p means p is not a “simple root”. Definition (Root multiplicity) A solution p of f ( x ) is a root (zero) of multiplicity m if f ( x ) = ( x − p ) m q ( x ) for some q s.t. lim x → p q ( x ) � = 0. Definition (Simple root) p is a simple root (zero) of f if its multiplicity m = 1. Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 65

Multiple roots Theorem (S.N.C. for simple root) f ∈ C 1 [ a , b ] has a simple root p ∈ ( a , b ) iff f ( p ) = 0 and f ′ ( p ) � = 0 . Proof. “= ⇒ ”: f ( x ) = ( x − p ) q ( x ) where lim x → p q ( x ) � = 0. Then f ′ ( x ) = q ( x ) + ( x − p ) q ′ ( x ). So f ∈ C 1 implies f ′ ( p ) = lim x → p f ′ ( x ) = lim x → p ( q ( x ) + ( x − p ) q ′ ( x )) � = 0 “ ⇐ =”: f ( x ) = f ( p ) + f ′ ( ξ ( x ))( x − p ) where ξ ( x ) between x and p . Define q ( x ) = f ′ ( ξ ( x )) then x → p f ′ ( ξ ( x )) = f ′ ( lim x → p ξ ( x )) = f ′ ( p ) � = 0 x → p q ( x ) = lim lim So f has a simple root at p . Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 66

Multiple roots Theorem (S.N.C. for multiple root) f ∈ C m [ a , b ] has a zero p of multiplicity m iff f ( p ) = f ′ ( p ) = · · · = f ( m − 1) ( p ) = 0 f ( m ) ( p ) � = 0 and Proof. Hint: Follow the proof above and use n � n � ( uv ) ( n ) = � u ( k ) v ( n − k ) k k =0 Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 67

Example Example (Multiple root) Let f ( x ) = e x − x − 1, show that f ( x ) has a zero of multiplicity 2 at x = 0. Solution. f ( x ) = e x − x − 1, f ′ ( x ) = e x − 1, and f ′′ ( x ) = e x . So f (0) = f ′ (0) = 0 and f ′′ (0) = 1 � = 0. By Theorem above f has root (zero) at x = 0 of multiplicity 2. Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 68

Modified Newton’s method Instead of using f ( x ) in Newton’s method, we can replace f by µ ( x ) := f ( x ) f ′ ( x ) We need to show: p is a root (simple or not) of f = ⇒ p is a simple root of µ Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 69

Modified Newton’s method Recall that f has a root p of multiplicity m if f ( x ) = ( x − p ) m q ( x ) for some q with lim x → p q ( x ) � = 0. Now there is ( x − p ) m q ( x ) µ ( x ) = f ( x ) f ′ ( x ) = m ( x − p ) m − 1 q ( x ) + ( x − p ) m q ′ ( x ) q ( x ) = ( x − p ) · mq ( x ) + ( x − p ) q ′ ( x ) q ( x ) mq ( x )+( x − p ) q ′ ( x ) → 1 where m � = 0 as x → p . By definition, µ ( x ) has simple root at p , i.e., µ ( p ) = 0 and µ ′ ( p ) � = 0. Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 70

Modified Newton’s method Now we use µ ( x ) instead of f ( x ) in Newton’s method: µ ′ ( x ) = x − ( f ( x ) / f ′ ( x )) f ( x ) f ′ ( x ) g ( x ) = x − µ ( x ) ( f ( x ) / f ′ ( x )) ′ = · · · = x − ( f ′ ( x )) 2 − f ( x ) f ′′ ( x ) The modified Newton’s method is f ( p n − 1 ) f ′ ( p n − 1 ) p n = p n − 1 − ( f ′ ( p n − 1 )) 2 − f ( p n − 1 ) f ′′ ( p n − 1 ) Drawbacks of the modified Newton’s method: ◮ Needs f ′′ in computation. ◮ Denominator approximates 0 as p n → p , so round-off may degrade convergence. Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 71

Accelerating convergence We showed that FPI generally has linear convergence only. How to improve? Suppose N is large, and p n , p n +1 , p n +2 satisfy p n +1 − p ≈ p n +2 − p p n − p p n +1 − p ( p n +1 − p ) 2 ≈ ( p n − p )( p n +2 − p ) = p n p n +2 − p ( p n +2 + p n ) + p 2 ⇐ ⇒ p n p n +2 − p 2 ( p n +1 − p n ) 2 n +1 ⇐ ⇒ p ≈ = · · · = p n − p n +2 − 2 p n +2 + p n p n +2 − 2 p n +1 + p n Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 72

Aitken’s ∆ 2 method Denote ∆ p n := p n +1 − p n , called forward difference , and ∆ 2 p n :=∆(∆ p n ) = ∆( p n +1 − p n ) =( p n +2 − p n +1 ) − ( p n +1 − p n ) = p n +2 − 2 p n +1 + p n So the result above can be written as p ≈ p n − (∆ p n ) 2 ∆ 2 p n . Aitken’s ∆ 2 method : p n = p n − (∆ p n ) 2 Given { p n } generated by FPI, set ˆ ∆ 2 p n . Then ˆ p n → p faster than p n . Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 73