Linear Algebra Review Leila Wehbe January 29, 2013 Leila Wehbe - PowerPoint PPT Presentation

Metrics Vector Spaces Banach Spaces Hilbert Space Matrices Linear Algebra Review Leila Wehbe January 29, 2013 Leila Wehbe Linear Algebra Review

Metrics Vector Spaces Banach Spaces Hilbert Space Matrices Table of contents Metrics Vector Spaces Banach Spaces Hilbert Space Matrices Leila Wehbe Linear Algebra Review

Metrics Vector Spaces Banach Spaces Hilbert Space Matrices Metric Given a space X , then d : X × X → R + 0 is a metric is for all x , y and z in X if: ◮ d ( x , y ) = 0 is equivalent to x = y ◮ d ( x , y ) = d ( y , x ) ◮ d ( x , y ) ≤ d ( x , z ) + d ( z , y ) Leila Wehbe Linear Algebra Review

Metrics Vector Spaces Banach Spaces Hilbert Space Matrices Example of a metric Euclidean Distance: n 1 Given X = R n , d ( x , y ) := ( ( x i − y i ) 2 ) � 2 i =1 ◮ d ( a , b ) = 0 is equivalent to a = b ◮ d ( a , b ) = d ( b , a ) ◮ d ( a , b ) ≤ d ( a , c ) + d ( c , b ) (this is the triangle inequality) Leila Wehbe Linear Algebra Review

Metrics Vector Spaces Banach Spaces Hilbert Space Matrices Vector Space A vector space is a space X such that for all x , y ∈ X and for all α ∈ R : ◮ x + y ∈ X ◮ α x ∈ X Leila Wehbe Linear Algebra Review

Metrics Vector Spaces Banach Spaces Hilbert Space Matrices Examples of vector spaces Real Numbers : given x , y ∈ R , and α ∈ R : ◮ x + y ∈ R ◮ α x ∈ R R n : given x , y ∈ R n , and α ∈ R : ◮ x + y ∈ R n ◮ α x ∈ R n Leila Wehbe Linear Algebra Review

Metrics Vector Spaces Banach Spaces Hilbert Space Matrices Examples of vector spaces n n a i x i and g ( x ) = b i x i , and � � Polynomials : given f ( x ) = i =0 i =0 α ∈ R : n ( a i + b i ) x i , i.e. polynomial of order n ◮ f ( x ) + g ( x ) = � i =0 n α a i x i , i.e. polynomial of order n ◮ α f ( x ) = � i =0 Leila Wehbe Linear Algebra Review

Metrics Vector Spaces Banach Spaces Hilbert Space Matrices Cauchy Series Given a space X , a Cauchy series is a series x i ∈ X for which for every ǫ > 0 there exist an n 0 such that for all m , n ≥ n 0 , d ( x m , x n ) ≤ ǫ Leila Wehbe Linear Algebra Review

Metrics Vector Spaces Banach Spaces Hilbert Space Matrices Completeness A space X is complete if the limit of every Cauchy series ∈ X . For example, (0 , 1) is not complete but [0 , 1] is. The set Q of rational numbers is not complete: you can construct √ √ a sequence that converges to 2 but 2 is not in Q . Leila Wehbe Linear Algebra Review

Metrics Vector Spaces Banach Spaces Hilbert Space Matrices Norm Given a vector space X , a norm is a mapping || . || : X → R + 0 that satisfies, for all x , y ∈ X and for all α ∈ R : ◮ || x || = 0 if and only if x = 0 ◮ || α x || = | α ||| x || ◮ || x + y || ≤ || x || + || y || (triangle inequality) A norm is also a metric: d ( x , y ) := || x − y || Leila Wehbe Linear Algebra Review

Metrics Vector Spaces Banach Spaces Hilbert Space Matrices Banach Space A Banach Space is a complete vector space X together with a norm || . || . � m � 1 p p Spaces : R m with the norm || x || := ℓ m | x i | p � i =1 � ∞ � 1 p ℓ p Spaces : These are subspaces of R N with || x || := � | x i | p i =1 � 1 p . �� X | f ( x ) | p dx Function Spaces L p ( X ): Over X , || f || := Leila Wehbe Linear Algebra Review

Metrics Vector Spaces Banach Spaces Hilbert Space Matrices Dot Product Given a vector space X , a dot product is a mapping � . , . � : X × X → R that satisfies, for all x , y and z ∈ X and for all α ∈ R : ◮ Symmetry: � x , y � = � y , x � ◮ Linearity: � x , α y � = α � x , y � ◮ Additivity: � x , y + z � = � x , y � + � x , z � Leila Wehbe Linear Algebra Review

Metrics Vector Spaces Banach Spaces Hilbert Space Matrices Hilbert Space A Hilbert Space is a complete vector space X together with a dot product � . , . � . � The dot product automatically generates a norm: || x || := � x , x � . Hilbert spaces are special cases of Banach spaces. Leila Wehbe Linear Algebra Review

Metrics Vector Spaces Banach Spaces Hilbert Space Matrices Examples of Hilbert Spaces Euclidean spaces and the standard dot product for x , y ∈ R m : m � � x , y � = x i y i i =1 Function spaces ( L 2 ( X )): functions on X with f : X → C for all � f , g ∈ F , with the dot product: � f , g � = X f ( x ) g ( x ) dx ℓ 2 series of real numbers (infinite), ∈ R N : ∞ � x , y � = � x i y i i =1 Leila Wehbe Linear Algebra Review

Metrics Vector Spaces Banach Spaces Hilbert Space Matrices Matrices A matrix M ∈ R m × n corresponds to a linear map from R m to R n . A symmetric matrix M ∈ R m × m satisfies M ij = M ji . An anti-symmetric matrix M ∈ R m × m satisfies M ij = − M ji . Rank: Denote by I the image of R m under M . rank( M ) is the smallest number of vectors that span I . Leila Wehbe Linear Algebra Review

Metrics Vector Spaces Banach Spaces Hilbert Space Matrices Matrices: orthogonality A matrix M ∈ R m × m is orthogonal if M T M = I . This means M T = M − 1 . An orthogonal matrix consists of mutually orthogonal rows and columns. Leila Wehbe Linear Algebra Review

Metrics Vector Spaces Banach Spaces Hilbert Space Matrices Matrix Norms The norm of a linear operator between two Banach spaces X and Y : || Ax || || A || := max || x || x ∈X || α Ax || ◮ || α A || = max = | α ||| A || || x || x ∈X || ( A + B ) x || || Ax || || Bx || ◮ || A + B || = max ≤ max || x || + max || x || = || x || x ∈X x ∈X x ∈X || A || + || B || || Ax || ◮ || A || = 0 implies max || x || and thus A x = 0 for all x , i.e. x ∈X A = 0. Leila Wehbe Linear Algebra Review

Metrics Vector Spaces Banach Spaces Hilbert Space Matrices Matrix Norms Frobenius norm: (in analogy with vector norm) m m || M || 2 M 2 Frob = � � ij i =1 j =1 Leila Wehbe Linear Algebra Review

Metrics Vector Spaces Banach Spaces Hilbert Space Matrices Eigen Systems Given M in R m × m , then λ ∈ R is an eigenvalue and x ∈ R m is an eigenvector if: M x = λ x Leila Wehbe Linear Algebra Review

Metrics Vector Spaces Banach Spaces Hilbert Space Matrices Eigen Systems, symmetric matrices For symmetric matrices all eigenvalues are real and the matrix is fully diagonalizable (i.e. m eigenvectors). All eigenvectors with different eigenvalues are mutually orthogonal: Proof, for two eigenvectors x and x ′ with respective eigenvalues λ and λ ′ : λ x T x ′ = ( M x ) T x = x T ( M T x ′ ) = x T ( M x ′ ) = λ ′ x T x ′ so λ ′ = λ or x T x = 0. We can decompose M = O T Λ O . Leila Wehbe Linear Algebra Review

Metrics Vector Spaces Banach Spaces Hilbert Space Matrices Eigen Systems, symmetric matrices We also have the operator norm: || Mx || 2 || M || 2 = max || x || 2 x ∈ R m x ∈ R m and || x || =1 || Mx || 2 = max x ∈ R m and || x || =1 x T M T M x = max x ∈ R m and || x || =1 x T O Λ O T O Λ O T x = max x ∈ R m and || x ′ || =1 x ′ T Λ 2 x ′ = max i ∈ [ m ] λ 2 = max i Leila Wehbe Linear Algebra Review

Metrics Vector Spaces Banach Spaces Hilbert Space Matrices Eigen Systems, symmetric matrices Frobenius norm: || M || 2 tr ( MM T ) = tr ( O Λ O T O Λ O T ) = Frob m tr (Λ O T O Λ O T O ) = tr (Λ 2 ) = � λ 2 = i i =1 Leila Wehbe Linear Algebra Review

Metrics Vector Spaces Banach Spaces Hilbert Space Matrices Matrices: Invariants m � Trace: tr ( M ) = M ii . i =1 tr ( AB ) = tr ( BA ). For symmetric matrices: m tr ( M ) = tr ( O T Λ O ) = tr (Λ OO T ) = tr (Λ) = � λ i i =1 Determinant: m � det ( M ) = λ i i =1 Leila Wehbe Linear Algebra Review

Metrics Vector Spaces Banach Spaces Hilbert Space Matrices Positive Matrices A Positive Definite Matrix is a matrix M ∈ R m × m for which for all x ∈ R m : x T M x > 0 if x � = 0 This matrix has only positive eigenvalues: x T M x = λ x T x = λ || x || > 0 Induced norm: || x || 2 M = x T M x Leila Wehbe Linear Algebra Review

Metrics Vector Spaces Banach Spaces Hilbert Space Matrices Singular Value Decomposition Want to find similar thing for arbitrary matrix M ∈ R m × n where m ≥ n : M = U Λ O U ∈ R m × n , U T U = I O ∈ R n × n , O T O = I Λ = diag ( λ 1 , λ 2 , ...λ n ) Leila Wehbe Linear Algebra Review