Section 3 Iterative Methods in Matrix Algebra Numerical Analysis II - PowerPoint PPT Presentation

Section 3 Iterative Methods in Matrix Algebra Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 155

Vector norm Definition A vector norm on R n , denoted by � · � , is a mapping from R n to R such that ◮ � x � ≥ 0 for all x ∈ R n , ◮ � x � = 0 if and only if x = 0 , ◮ � α x � = | α |� x � for all α ∈ R and x ∈ R n , ◮ � x + y � ≤ � x � + � y � for all x , y ∈ R . Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 156

Vector norm Definition ( l p norms) The l p (sometimes L p or ℓ p ) norm of a vector is defined by | x i | p � 1 / p � n � 1 ≤ p < ∞ : � x � p = i =1 p = ∞ : � x � ∞ = max 1 ≤ i ≤ n | x i | In particular, the l 2 norm is also called the Euclidean norm . Note that when 0 ≤ p < 1, � · � p is not norm, strictly speaking, but have some usages in specific applications. Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 157

l 2 norm x 3 x 2 The vectors in the first octant of � 3 The vectors in � 2 with l 2 norm less with l 2 norm less than 1 are inside (0, 1) than 1 are inside this figure. this figure. (0, 0, 1) ( � 1, 0) (1, 0) x 1 (0, 1, 0) (1, 0, 0) x 2 x 1 (0, � 1) Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 158

l ∞ norm x 2 x 3 (0, 1) (1, 1) (0, 0, 1) ( � 1, 1) (1, 0, 1) (0, 1, 1) (1, 1, 1) x 1 ( � 1, 0) (1, 0) (1, 0, 0) (0, 1, 0) x 1 x 2 (1, 1, 0) ( � 1, � 1) (0, � 1) (1, � 1) The vectors in � 2 with The vectors in the first l � norm less than 1 are octant of � 3 with l � norm inside this figure. less than 1 are inside this figure. Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 159

Vector norms Example Compute the l 2 and l ∞ norms of vector x = (1 , − 1 , 2) ∈ R 3 . Solution: √ � | 1 | 2 + | − 1 | 2 + | 2 | 2 = � x � 2 = 6 � x � ∞ = max 1 ≤ i ≤ 3 | x i | = max {| 1 | , | − 1 | , | 2 |} = 2 Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 160

Theorem (Cauchy-Schwarz inequality) For any vectors x = ( x 1 , . . . , x n ) ⊤ ∈ R n and y = ( y 1 , . . . , y n ) ⊤ ∈ R n , there is | x i | 2 � 1 / 2 � n | y i | 2 � 1 / 2 n � n � � � � � | x ⊤ y | = x i y i � ≤ = � x � 2 � y � 2 � � � i =1 i =1 i =1 Proof. It is obviously true for x = 0 or y = 0. If x , y � = 0, then for any λ ∈ R , there is 0 ≤ � x − λ y � 2 2 = � x � 2 2 − 2 λ x ⊤ y + λ 2 � y � 2 2 and the equality holds when λ = � x � 2 / � y � 2 . Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 161

Distance between vectors Definition (Distance between two vectors) The l p distance ( 1 ≤ p ≤ ∞ ) between two vectors x , y ∈ R n is defined by � x − y � p . Definition (Convergence of a sequence of vectors) A sequence { x ( k ) } is said to converge with respect to the l p norm if for any given ǫ > 0 , there exists an integer N ( ǫ ) such that � x ( k ) − x � < ǫ, for all k ≥ N ( ǫ ) Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 162

Convergence of a sequence of vectors Theorem A sequence of vectors { x ( k ) } converges to x if and only if x ( k ) → x i for every i = 1 , 2 , . . . , n. i Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 163

Theorem For any vector x ∈ R n , there is � x � ∞ ≤ � x � 2 ≤ √ n � x � ∞ Proof. � | x i | 2 ≤ | x 1 | 2 + · · · + | x n | 2 = � x � 2 � � x � ∞ = max | x i | = max i i � | x 1 | 2 + · · · + | x n | 2 ≤ � | x i | 2 � x � 2 = n max i = √ n | x i | 2 = √ n max | x i | = √ n � x � ∞ � max i i Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 164

Compare l 2 and l ∞ norms in R 2 x 2 � x � � � 1 1 � x � 2 � 1 � 1 1 x 1 √ 2 � x � � � 2 � 1 Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 165

Matrix norm Definition A matrix norm on the set of n × n matrices is a real-valued function, denoted by � · � , that satisfies the follows for all A , B ∈ R n × n and α ∈ R : ◮ � A � ≥ 0 ◮ � A � = 0 if and only if A = 0 the zero matrix, ◮ � α A � = | α |� A � ◮ � A + B � ≤ � A � + � B � ◮ � AB � ≤ � A �� B � Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 166

Distance between matrices Definition Suppose � · � is a norm defined on R n × n . Then the distance between two n × n matrices A and B with respect to � · � is � A − B � (check that it’s a distance) Matrix norm can be induced by vector norms, and hence there are many choices. Here we focus on those induced by l 2 and l ∞ vector norms. Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 167

Matrix norm Definition If � · � is a vector norm on R n , then the norm defined below � A � = max � x � =1 � Ax � is called the matrix norm induced by vector norm � · � . Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 168

Matrix norm Remark ◮ Induced norms are also called natural norms of matrices. ◮ Unless otherwise specified, by matrix norms most books/papers refer to induced norms. ◮ The induced norm can be written equivalently as � Ax � � A � = max � x � x � =0 ◮ It can be easily extended to case A ∈ R m × n . Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 169

Matrix norm Corollary For any vector x ∈ R n , there is � Ax � ≤ � A �� x � . Proof. It is obvious for x = 0. If x � = 0, then � Ax ′ � � Ax � � x � ≤ max � x ′ � = � A � x ′ � =0 Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 170

Induced l 2 matrix norm x 2 3 A x for � x � 2 � 1 x 2 A x � x � 2 � 1 1 1 � A � 2 � 1 x x 1 x 1 1 � 2 � 1 1 2 � 1 � 1 � 3 Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 171

Induced l ∞ matrix norm x 2 x 2 A x for � x � � � 1 2 � A � � � x � � � 1 1 1 x A x 1 x 1 x 1 � 1 � 2 � 1 1 2 � 1 � 1 � 2 Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 172

Matrix norm Theorem � n Suppose A = [ a ij ] ∈ R n × n , then � A � ∞ = max 1 ≤ i ≤ n j =1 | a ij | . Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 173

Matrix norm Proof. For any x with � x � ∞ = 1, i.e., max i | x i | = 1, there is � � � � � � � � � Ax � ∞ = max a 1 j x j a nj x j � � � � � , . . . , � � � j j �� � � ≤ max | a 1 j || x j | , . . . , | a nj || x j | j j n �� � � � ≤ max | a 1 j | , . . . , | a nj | = max | a ij | 1 ≤ i ≤ n j j j =1 Suppose i ′ is such that � n � n j =1 | a i ′ j | = max 1 ≤ i ≤ n j =1 | a ij | , then by choosing ˆ x such that ˆ x j = 1 if a i ′ j ≥ 0 and − 1 otherwise, we have � n x j = � n � n j =1 a i ′ j ˆ j =1 | a i ′ j | . So � A ˆ x � ∞ = max 1 ≤ i ≤ n j =1 | a ij | . Note � n that � ˆ x � ∞ = 1. Therefore � A � ∞ = max 1 ≤ i ≤ n j =1 | a ij | . Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 174

Eigenvalues and eigenvectors of square matrices Definition The characteristic polynomial of a square matrix A ∈ R n × n is defined by p ( λ ) = det( A − λ I ) We call λ an eigenvalue of A if λ is a root of p, i.e., det( A − λ I ) = 0 . Moreover, any nonzero solution x ∈ R n of ( A − λ I ) x = 0 is called an eigenvector of A corresponding to the eigenvalue λ . Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 175

Eigenvalues and eigenvectors of square matrices Remark ◮ p ( λ ) is a polynomial of degree n, and hence has n roots. ◮ x is an eigenvector of A corresponding to eigenvalue λ iff ( A − λ I ) x = 0 , i.e., Ax = λ x. This also means A applied to x is stretching x by λ . ◮ If x is an eigenvector of A corresponding to λ , so is α x for any α � = 0 : A ( α x ) = α Ax = αλ x = λ ( α x ) Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 176

Eigenvalues and eigenvectors of square matrices Definition Let λ 1 , . . . , λ n be the eigenvalues of A ∈ R n × n , then the spectral radius ρ ( A ) is defined by ρ ( A ) = max i | λ i | where | · | is the absolute value (aka magnitude) of complex numbers. Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 177