#5 Norms L. Olson October 1, 2015 Department of Computer Science - PowerPoint PPT Presentation

#5 Norms L. Olson October 1, 2015 Department of Computer Science University of Illinois at Urbana-Champaign 1

objectives • Set up an array of data and measure its “size” • Construct a “norm” and apply its properties to a problem • Describe a “matrix norm” or “operator norm” • Find examples where a matrix norm is appropriate and not appropriate 2

vector addition and subtraction Addition and subtraction are element-by-element operations ⇐⇒ c = a + b c i = a i + b i i = 1 , . . . , n d = a − b ⇐⇒ d i = a i − b i i = 1 , . . . , n     1 3 a = 2 b = 2         3 1     4 − 2 a + b = a − b = 4 0         4 2 3

multiplication by a scalar Multiplication by a scalar involves multiplying each element in the vector by the scalar: ⇐⇒ b = σ a b i = σ a i i = 1 , . . . , n     4 2 b = a a = 6 2 = 3         8 4 4

vector transpose The transpose of a row vector is a column vector:   1 u T = � � u = 1 , 2 , 3 then 2     3 Likewise if v is the column vector   4 v T = � � v = then 4 , 5 , 6 5     6 5

linear combinations Combine scalar multiplication with addition         u 1 v 1 α u 1 + β v 1 w 1 u 2 v 2 α u 2 + β v 2 w 2                 α + β = = . . . .         . . . .  .   .   .   .          u m v m α u m + β v m w m     − 2 1 r = 1 s = 0         3 3       − 4 3 − 1 t = 2 r + 3 s =  +  = 2 0 2           6 9 15 6

linear combinations Any one vector can be created from an infinite combination of other “suitable” vectors. � � � � � � 4 1 0 w = = 4 + 2 2 0 1 � � � � 1 1 w = 6 − 2 0 − 1 � � � � 2 − 1 w = − 2 4 1 � � � � � � 4 1 0 w = 2 − 4 − 2 2 0 1 7

linear combinations Graphical 4 [2,4] interpretation: [1,-1] 3 • Vector tails can be moved to 2 [4,2] convenient [-1,1] locations 1 [0,1] • Magnitude and [1,1] 0 direction of vectors [1,0] is preserved 0 1 2 3 4 5 6 8

vector inner product In physics, analytical geometry, and engineering, the dot product has a geometric interpretation n � σ = x · y ⇐⇒ σ = x i y i i = 1 x · y = � x � 2 � y � 2 cos θ 9

vector inner product The inner product of x and y requires that x be a row vector y be a column vector   y 1 y 2   � � x 1 x 2 x 3 x 4  = x 1 y 1 + x 2 y 2 + x 3 y 3 + x 4 y 4    y 3   y 4 10

vector inner product For two n -element column vectors, u and v , the inner product is n � σ = u T v ⇐⇒ σ = u i v i i = 1 The inner product is commutative so that (for two column vectors) u T v = v T u 11

vector outer product The inner product results in a scalar. The outer product creates a rank-one matrix: A = uv T ⇐⇒ a i , j = u i v j Example Outer product of two 4-element column vectors   u 1 u 2 uv T =   � � v 1 v 2 v 3 v 4   u 3     u 4   u 1 v 1 u 1 v 2 u 1 v 3 u 1 v 4 u 2 v 1 u 2 v 2 u 2 v 3 u 2 v 4 12   =   u v u v u v u v  

vector norms Compare magnitude of scalars with the absolute value � > � � � � � α � β � Compare magnitude of vectors with norms � x � > � y � There are several ways to compute || x || . In other words the size of two vectors can be compared with different norms. 13

vector norms Consider two element vectors, which lie in a plane b = (2,4) a = (4,2) c = (2,1) a = (4,2) Use geometric lengths to represent the magnitudes of the vectors √ √ √ � 4 2 + 2 2 = � 2 2 + 4 2 = � 2 2 + 1 2 = ℓ a = 20 , ℓ b = 20 , ℓ c = 5 We conclude that ℓ a = ℓ b and ℓ a > ℓ c or � a � = � b � and � a � > � c � 14

the l 2 norm The notion of a geometric length for 2D or 3D vectors can be extended vectors with arbitrary numbers of elements. The result is called the Euclidian or L 2 norm: � n � 1 / 2 � � 1 / 2 = � x 2 1 + x 2 2 + . . . + x 2 x 2 � x � 2 = n i i = 1 The L 2 norm can also be expressed in terms of the inner product √ √ � x � 2 = x · x = x T x 15

p -norms For any positive integer p | x 1 | p + | x 2 | p + . . . + | x n | p � 1 / p � � x � p = The L 1 norm is sum of absolute values n � � x � 1 = | x 1 | + | x 2 | + . . . + | x n | = | x i | i = 1 The L ∞ norm or max norm is � x � ∞ = max ( | x 1 | , | x 2 | , . . . , | x n | ) = max ( | x i | ) i Although p can be any positive number, p = 1 , 2 , ∞ are most commonly used. 16

defining a p -norm These must hold for any x and y 1. � x � > 0 if x � 0 2. � α x � = | α | · � x � for an scalar α 3. � x + y � � � x � + � y � (this is called the triangle inequality) 17

defining a p -norm for a matrix If A is a matrix, then we use the vector p -norm to define a similar matrix norm: � Ax � p � A � p = max � x � p x � 0 18

application of norms Are two vectors (nearly) equal? Floating point comparison of two scalars with absolute value: � � � α − β � < δ � � � α � where δ is a small tolerance. Comparison of two vectors with norms: � y − z � < δ � z � 19

application of norms Notice that � y − z � < δ � z � is not equivalent to � y � − � z � < δ . � z � This comparison is important in convergence tests for sequences of vectors. 20

application of norms Creating a Unit Vector Given u = [ u 1 , u 2 , . . . , u m ] T , the unit vector in the direction of u is u u = ˆ � u � 2 Proof: � u � 1 � � � ˆ u � 2 = = � u � 2 = 1 � � � u � 2 � u � 2 � � 2 The following are not unit vectors u u � u � 1 � u � ∞ 21

orthogonal vectors From geometric interpretation of the inner product u · v = � u � 2 � v � 2 cos θ u T v u · v cos θ = = � u � 2 � v � 2 � u � 2 � v � 2 Two vectors are orthogonal when θ = π/ 2 or u · v = 0. In other words u T v = 0 if and only if u and v are orthogonal . 22

orthonormal vectors Orthonormal vectors are unit vectors that are orthogonal . A unit vector has an L 2 norm of one. The unit vector in the direction of u is u u = ˆ � u � 2 Since √ � u � 2 = u · u it follows that u · u = 1 if u is a unit vector. 23

notation The matrix A with m rows and n columns looks like:   a 11 a 12 · · · a 1 n a 21 a 22 a 2 n     A = . .   . . . .     a m 1 · · · a mn a ij = element in row i , and column j 24

matrices consist of row and column vectors As a collection of column vec- As a collection of row vectors tors a ′   ( 1 ) � � �     � � �   � � �    � � �  a ′    � � �  ( 2 ) A = a ( 1 ) a ( 2 ) · · · a ( n )    � � �     � � �  A =    � � �   .   � � �  .   � � � .   � � �       a ′ ( m ) A prime is used to designate a row vector on this and the fol- lowing pages. 25

preview of the row and column view Matrix and vector operations � Row and column operations � Element-by-element operations 26

matrix operations Addition and subtraction C = A + B or c i , j = a i , j + b i , j i = 1 , . . . , m ; j = 1 , . . . , n Multiplication by a Scalar B = σ A or b i , j = σ a i , j i = 1 , . . . , m ; j = 1 , . . . , n Note Commas in subscripts are necessary when the subscripts are as- signed numerical values. For example, a 2 , 3 is the row 2, column 3 element of matrix A , whereas a 23 is the 23 rd element of vector a . 27 When variables appear in indices, such as a ij or a i , j , the comma is

matrix transpose B = A T or b i , j = a j , i i = 1 , . . . , m ; j = 1 , . . . , n 28

matrix–vector product • The Column View • gives mathematical insight • The Row View • easy to do by hand • The Vector View • A square matrix rotates and stretches a vector 29

column view of matrix–vector product Consider a linear combination of a set of column vectors { a ( 1 ) , a ( 2 ) , . . . , a ( n ) } . Each a ( j ) has m elements Let x i be a set (a vector) of scalar multipliers x 1 a ( 1 ) + x 2 a ( 2 ) + . . . + x n a ( n ) = b or n � a ( j ) x j = b j = 1 Expand the (hidden) row index         a 11 a 12 a 1 n b 1 a 21 a 22 a 2 n b 2                 x 1 + x 2 + · · · + x n = . . . .         . . . .  .   .   .   .          a m 1 a m 2 a mn b m 30