ECS231 Mathematics Review I: Linear Algebra Reference: Chap.1 of - PowerPoint PPT Presentation

ECS231 Mathematics Review I: Linear Algebra Reference: Chap.1 of Solomon 1 / 23

Vector spaces over R Denote a (abstract) vector by v . A vector space V = { a collection of vectors v } which satisfies ◮ All v, w ∈ V can be added and multiplied by α ∈ R : v + w ∈ V , α · v ∈ V ◮ The operations ‘+ , · ’ must satisfy the axioms : For arbitrary u, v, w ∈ V , 1. ‘+’ commutativity and associativity: v + w = w + v , ( u + v ) + w = u + ( v + w ). 2. Distributivity: α ( v + w ) = αv + αw , ( α + β ) v = αv + βv , for all α, β ∈ R . 3. ‘+’ identity: there exists 0 ∈ V with 0 + v = v . 4. ‘+’ inverse: for any v ∈ V , there exists w ∈ V with v + w = 0. 5. ‘ · ’ identity: 1 · v = v . 6. ‘ · ’ compatibility: for all α, β ∈ R , ( αβ ) · v = α · ( β · v ). 2 / 23

Example ◮ Euclidean space: � � R n = a ≡ ( a 1 , a 2 , . . . , a n ): a i ∈ R . ◮ Addition: ( a 1 , . . . , a n ) + ( b 1 , . . . , b n ) = ( a 1 + b 1 , . . . , a n + b n ) ◮ Multiplication: c · ( a 1 , . . . , a n ) = ( ca 1 , . . . , ca n ) ◮ Illustration in R 2 : a + b 2 a b a a 3 / 23

Example ◮ Polynomials: � � � a i x i : a i ∈ R R [ x ] = p ( x ) = . i ◮ Addition and multiplication in the usual way, e.g. p ( x ) = a 0 + a 1 x + a 2 x 2 , q ( x ) = b 1 x : ◮ Addition: p ( x ) + q ( x ) = a 0 + ( a 1 + b 1 ) x + a 2 x 2 . ◮ Multiplication: 2 p ( x ) = 2 a 0 + 2 a 1 x + 2 a 2 x 2 . 4 / 23

Span of vectors ◮ Start with v 1 , . . . , v n ∈ V , and a i ∈ R , we can define n � v ≡ a i v i = a 1 v 1 + a 2 v 2 + · · · + a n v n , i =1 Such a v is called a linear combination of v 1 , . . . , v n . ◮ For a set of vectors S = { v i : i ∈ I} , all its linear combinations define � � � span S ≡ a i v i : v i ∈ S and a i ∈ R i 5 / 23

Example in R 2 ◮ Observation from (c): adding a new vector does not always increase the span. 6 / 23

Linear dependence ◮ A set S of vectors is linearly dependent if it contains a vector k � v = for some v i ∈ S \{ v } and nonzero c i ∈ R . c i v i , i =1 ◮ Otherwise, S is called linearly independent . ◮ Two other equivalent defs. of linear dependence: ◮ There exists { v 1 , . . . , v k } ⊂ S \{ 0 } such that k � where c i � = 0 for all i . c i v i = 0 i =1 ◮ There exists v ∈ S such that span S = span( S \{ v } ) . 7 / 23

Dimension and basis ◮ Given a vector space V , it is natural to build a finite set of linearly independent vectors: { v 1 , . . . , v n } ⊂ V . ◮ The max number n of such vectors defines the dimension of V . ◮ Any set S of such vectors is a basis of V , and satisfies span S = V . 8 / 23

Examples ◮ The standard basis for R n is given by the n vectors e i = (0 , . . . , 0 , 1 , 0 , . . . , 0 ) for i = 1 , . . . , n � �� � � �� � i − 1 n − i Since ◮ e i is not linear combination of the rest of vectors. ◮ For all c ∈ R n , we have c = � n i =1 c i e i . Hence, the dimension of R n is n . ◮ A basis of polynomials R [ x ] is given by monomials { 1 , x, x 2 , . . . } . The dimension of R [ x ] is ∞ . 9 / 23

More about R n ◮ Dot product: for a = ( a 1 , . . . , a n ) , b = ( b 1 , . . . , b n ) ∈ R n n � a · b = a i b i . i =1 ◮ Length of a vector � n = √ a · a. a 2 1 + · · · + a 2 � a � 2 = ◮ Angle between two vectors a · b θ = arccos . � a � 2 � b � 2 (*Motivating trigonometric in R 3 : a · b = � a � 2 � b � 2 cos θ .) ◮ Vectors a, b are orthogonal if a · b = 0 = cos 90 ◦ . 10 / 23

Linear function ◮ Given two vector spaces V , V ′ , a function L : V → V ′ is linear , if it preserves linearity . ◮ Namely, for all v 1 , v 2 ∈ V and c ∈ R , ◮ L [ v 1 + v 2 ] = L [ v 1 ] + L [ v 2 ]. ◮ L [ cv 1 ] = c L [ v 1 ]. ◮ L is completely defined by its action on a basis of V : � L [ v ] = c i L [ v i ] , i where v = � i c i v i and { v 1 , v 2 , . . . } is a basis of V . 11 / 23

Examples ◮ Linear map in R n : L : R 2 → R 3 defined by L [( x, y )] = (3 x, 2 x + y, − y ) . ◮ Integration operator: linear map L : R [ x ] → R [ x ] defined by � 1 L [ p ( x )] = p ( x ) dx. 0 12 / 23

Matrix ◮ Write vectors in R m in ‘ column forms ’, e.g.,       v 11 v 12 v 1 n . . .       . . . v 1 =  , v 2 =  , . . . , v n =  . . . .    v m 1 v m 2 v mn ◮ Put n columns together we obtain an m × n matrix   v 11 v 12 . . . v 1 n   | | |  v 21 v 22 . . . v 2 n   =   V ≡ v 1 v 2 . . . v n . . . .   . . . .  . . . .   | | | v m 1 v m 2 . . . v mn ◮ The space of all such matrices is denoted by R m × n . 13 / 23

Unified notation: Scalars, Vectors, and Matrices ◮ A scalar c ∈ R is viewed as a 1 × 1 matrix c ∈ R 1 × 1 . ◮ A column vector v ∈ R n is viewed as an n × 1 matrix v ∈ R n × 1 . 14 / 23

Matrix vector multiplication ◮ A matrix V ∈ R m × n can be multiplied by a vector c ∈ R n :     c 1 | | | .  .  v 1 v 2 . . . v n  = c 1 v 1 + c 2 v 2 + · · · + c n v n .   .  | | | c n V c is a linear combination of the columns of V . This is fundamental. 15 / 23

Using matrix notation ◮ Matrix vector multiplication can be denoted by = A x b . ���� ���� ���� R m × n R n R m ◮ M ∈ R m × n multiplied by another matrix in R n × k can be defined as M [ c 1 , . . . , c k ] ≡ [ Mc 1 , . . . , Mc k ] . 16 / 23

Example ◮ Identity matrix   1 0 0 . . .   | | | . ... .   0 1 .  =   I n ≡ e 1 e 2 . . . e n  .  .  ... ...  . . 0  | | | 0 0 1 . . . It holds I n c = c for all c ∈ R n . 17 / 23

Example ◮ Linear map L [( x, y )] = (3 x, 2 x + y, − y ) satisfies     3 0 3 x � x � L [( x, y )] = 2 1 · = 2 x + y .     y − 1 − y 0 ���� � �� � � �� � R 2 R 3 × 2 R 3 ◮ All linear maps L : R n → R m can be expressed as L [ x ] = Ax, for some matrix A ∈ R m × n . 18 / 23

Matrix transpose ◮ Use A ij to denote the element of A at row i column j . ◮ The transpose of A ∈ R m × n is defined as A T ∈ R n × m ( A T ) ij = A ji . Example:   1 2 � 1 � 3 5 A T = A = 3 4 ⇒ .   2 4 6 5 6 ◮ Basic identities: ( A T ) T = A, ( A + B ) T = A T + B T , ( AB ) T = B T A T . 19 / 23

Examples: Matrix operations with transpose ◮ Dot product of a, b ∈ R n :   b 1 n � . � �  .  = a T b.  a · b = a i b i = a 1 . . . a n .  i =1 b n ◮ Residual norms of r = Ax − b : � Ax − b � 2 2 = ( Ax − b ) T ( Ax − b ) = ( x T A T − b T )( Ax − b ) = b T b − b T Ax − x T A T b + x T A T Ax (by b T Ax = x T A T b ) = � b � 2 2 − 2 b T Ax + � Ax � 2 2 . 20 / 23

Computation aspects ◮ Storage of matrices in memory:    Row-major: 1 2 3 4 5 6 1 2    ⇒ 3 4   5 6  Column-major: 1 3 5 2 4 6 ◮ Multiplication b = Ax for A ∈ R m × n and x ∈ R n : Access A row-by-row: Access column-by-column: 1: b = 0 1: b = 0 2: for i = 1 , . . . , m do 2: for j = 1 , . . . , n do 3: for j = 1 , . . . , n do 3: for i = 1 , . . . , m do b i = b i + A ij x j b i = b i + A ij x j 4: 4: 5: end for 5: end for 6: end for 6: end for 21 / 23

Linear systems of equations in matrix form ◮ Example : find ( x, y, z ) satisfying       3 x + 2 y + 5 z = 0 3 2 5 x 0  = ⇒ − 4 x + 9 y − 3 z = − 7 − 4 9 − 3 − 7 y      2 − 3 − 3 z 1 2 x − 3 y − 3 z = 1 . ◮ Given A = [ a 1 , . . . , a n ] ∈ R m × n , b ∈ R m , find x ∈ R n : Ax = b. ◮ Solution exists if b is in column space of A : � n � � b ∈ col A ≡ { Ax : x ∈ R n } = x i a i : x i ∈ R . i =1 The dimension of col A is defined as the rank of A . 22 / 23

The square case ◮ Let A ∈ R n × n be a square matrix, and suppose Ax = b has solution for all b ∈ R n . We can solve Ax i = e i , for i = 1 , . . . , n. � � � = I n A x 1 x 2 . . . x n � �� � A − 1 ◮ The inverse satisfies ( why ?) AA − 1 = A − 1 A = I n ( A − 1 ) − 1 = A. and ◮ Hence, for any b , we can express the solution as x = A − 1 Ax = A − 1 b. 23 / 23