Functional Analysis Review Lorenzo Rosasco slides courtesy of Andre - PowerPoint PPT Presentation

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Functional Analysis Review Lorenzo Rosasco –slides courtesy of Andre Wibisono 9.520: Statistical Learning Theory and Applications February 13, 2012 L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators 1 Vector Spaces 2 Hilbert Spaces 3 Matrices 4 Linear Operators L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Vector Space A vector space is a set V with binary operations +: V × V → V and · : R × V → V such that for all a , b ∈ R and v , w , x ∈ V : 1 v + w = w + v 2 ( v + w ) + x = v + ( w + x ) 3 There exists 0 ∈ V such that v + 0 = v for all v ∈ V 4 For every v ∈ V there exists − v ∈ V such that v + (− v ) = 0 5 a ( bv ) = ( ab ) v 6 1 v = v 7 ( a + b ) v = av + bv 8 a ( v + w ) = av + aw L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Vector Space A vector space is a set V with binary operations +: V × V → V and · : R × V → V such that for all a , b ∈ R and v , w , x ∈ V : 1 v + w = w + v 2 ( v + w ) + x = v + ( w + x ) 3 There exists 0 ∈ V such that v + 0 = v for all v ∈ V 4 For every v ∈ V there exists − v ∈ V such that v + (− v ) = 0 5 a ( bv ) = ( ab ) v 6 1 v = v 7 ( a + b ) v = av + bv 8 a ( v + w ) = av + aw Example: R n , space of polynomials, space of functions. L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Inner Product An inner product is a function �· , ·� : V × V → R such that for all a , b ∈ R and v , w , x ∈ V : L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Inner Product An inner product is a function �· , ·� : V × V → R such that for all a , b ∈ R and v , w , x ∈ V : 1 � v , w � = � w , v � 2 � av + bw , x � = a � v , x � + b � w , x � 3 � v , v � � 0 and � v , v � = 0 if and only if v = 0. L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Inner Product An inner product is a function �· , ·� : V × V → R such that for all a , b ∈ R and v , w , x ∈ V : 1 � v , w � = � w , v � 2 � av + bw , x � = a � v , x � + b � w , x � 3 � v , v � � 0 and � v , v � = 0 if and only if v = 0. v , w ∈ V are orthogonal if � v , w � = 0. L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Inner Product An inner product is a function �· , ·� : V × V → R such that for all a , b ∈ R and v , w , x ∈ V : 1 � v , w � = � w , v � 2 � av + bw , x � = a � v , x � + b � w , x � 3 � v , v � � 0 and � v , v � = 0 if and only if v = 0. v , w ∈ V are orthogonal if � v , w � = 0. Given W ⊆ V , we have V = W ⊕ W ⊥ , where W ⊥ = { v ∈ V | � v , w � = 0 for all w ∈ W } . L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Inner Product An inner product is a function �· , ·� : V × V → R such that for all a , b ∈ R and v , w , x ∈ V : 1 � v , w � = � w , v � 2 � av + bw , x � = a � v , x � + b � w , x � 3 � v , v � � 0 and � v , v � = 0 if and only if v = 0. v , w ∈ V are orthogonal if � v , w � = 0. Given W ⊆ V , we have V = W ⊕ W ⊥ , where W ⊥ = { v ∈ V | � v , w � = 0 for all w ∈ W } . Cauchy-Schwarz inequality: � v , w � � � v , v � 1 / 2 � w , w � 1 / 2 . L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Norm Can define norm from inner product: � v � = � v , v � 1 / 2 . L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Norm A norm is a function � · � : V → R such that for all a ∈ R and v , w ∈ V : 1 � v � � 0, and � v � = 0 if and only if v = 0 2 � av � = | a | � v � 3 � v + w � � � v � + � w � Can define norm from inner product: � v � = � v , v � 1 / 2 . L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Metric Can define metric from norm: d ( v , w ) = � v − w � . L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Metric A metric is a function d : V × V → R such that for all v , w , x ∈ V : 1 d ( v , w ) � 0, and d ( v , w ) = 0 if and only if v = w 2 d ( v , w ) = d ( w , v ) 3 d ( v , w ) � d ( v , x ) + d ( x , w ) Can define metric from norm: d ( v , w ) = � v − w � . L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Basis B = { v 1 , . . . , v n } is a basis of V if every v ∈ V can be uniquely decomposed as v = a 1 v 1 + · · · + a n v n for some a 1 , . . . , a n ∈ R . L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Basis B = { v 1 , . . . , v n } is a basis of V if every v ∈ V can be uniquely decomposed as v = a 1 v 1 + · · · + a n v n for some a 1 , . . . , a n ∈ R . An orthonormal basis is a basis that is orthogonal ( � v i , v j � = 0 for i � = j ) and normalized ( � v i � = 1). L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators 1 Vector Spaces 2 Hilbert Spaces 3 Matrices 4 Linear Operators L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Hilbert Space, overview Goal: to understand Hilbert spaces (complete inner product spaces) and to make sense of the expression ∞ � f = � f , φ i � φ i , f ∈ H i = 1 Need to talk about: 1 Cauchy sequence 2 Completeness 3 Density 4 Separability L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Cauchy Sequence Recall: lim n → ∞ x n = x if for every ǫ > 0 there exists N ∈ N such that � x − x n � < ǫ whenever n � N . L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Cauchy Sequence Recall: lim n → ∞ x n = x if for every ǫ > 0 there exists N ∈ N such that � x − x n � < ǫ whenever n � N . ( x n ) n ∈ N is a Cauchy sequence if for every ǫ > 0 there exists N ∈ N such that � x m − x n � < ǫ whenever m , n � N . L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Cauchy Sequence Recall: lim n → ∞ x n = x if for every ǫ > 0 there exists N ∈ N such that � x − x n � < ǫ whenever n � N . ( x n ) n ∈ N is a Cauchy sequence if for every ǫ > 0 there exists N ∈ N such that � x m − x n � < ǫ whenever m , n � N . Every convergent sequence is a Cauchy sequence (why?) L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Completeness A normed vector space V is complete if every Cauchy sequence converges. L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Completeness A normed vector space V is complete if every Cauchy sequence converges. Examples: 1 Q is not complete. 2 R is complete (axiom). 3 R n is complete. 4 Every finite dimensional normed vector space (over R ) is complete. L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Hilbert Space A Hilbert space is a complete inner product space. L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Hilbert Space A Hilbert space is a complete inner product space. Examples: 1 R n 2 Every finite dimensional inner product space. n = 1 | a n ∈ R , � ∞ 3 ℓ 2 = { ( a n ) ∞ n = 1 a 2 n < ∞ } � 1 0 f ( x ) 2 dx < ∞ } 4 L 2 ([ 0, 1 ]) = { f : [ 0, 1 ] → R | L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Density Y is dense in X if Y = X . L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Density Y is dense in X if Y = X . Examples: 1 Q is dense in R . 2 Q n is dense in R n . 3 Weierstrass approximation theorem: polynomials are dense in continuous functions (with the supremum norm, on compact domains). L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Separability X is separable if it has a countable dense subset. L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Separability X is separable if it has a countable dense subset. Examples: 1 R is separable. 2 R n is separable. 3 ℓ 2 , L 2 ([ 0, 1 ]) are separable. L. Rosasco Functional Analysis Review

Outline Vector Spaces Hilbert Spaces Matrices Linear Operators Orthonormal Basis A Hilbert space has a countable orthonormal basis if and only if it is separable. Can write: ∞ � f = � f , φ i � φ i for all f ∈ H . i = 1 L. Rosasco Functional Analysis Review