CSE 446: Linear Algebra Review Sachin Mehta University of - PowerPoint PPT Presentation

CSE 446: Linear Algebra Review Sachin Mehta University of Washington, Seattle Email: sacmehta@uw.edu Sachin Mehta ( University of Washington, Seattle Email: sacmehta@uw.edu ) CSE 446: Linear Algebra Review 1 / 15

Things to get from Today Basics of Vector and Matrix operations Matrix Differentiation EigenValues and EigenVectors Sachin Mehta ( University of Washington, Seattle Email: sacmehta@uw.edu ) CSE 446: Linear Algebra Review 2 / 15

Vectors A vector v ∈ R n is an n-tuple of real numbers.   v 1  2  v 2   v = e . g . v = 3 .   .     . 5   v n � v 2 1 + v 2 2 + .. + v 2 Length of v is || v || = n Sachin Mehta ( University of Washington, Seattle Email: sacmehta@uw.edu ) CSE 446: Linear Algebra Review 3 / 15

Vector Operations Addition and Subtraction: To add or subtract two vectors, add or subtract them component wise     v 1 u 1 v 2 u 2     v ± u =  ±  .   .  . .     . .    v n u n Scaling: This is just like expanding or shrinking the vector. Let α be a scalar, then vector v after scaling with α can be represented as α v .   α v 1 α v 2 v ′ = α v =    .  .   .   α v n Sachin Mehta ( University of Washington, Seattle Email: sacmehta@uw.edu ) CSE 446: Linear Algebra Review 4 / 15

Vector Operations Inner (or dot) product of two vectors: Let u and v be two vectors, then their dot product is defined as: u . v = u T v       u 1 v 1 v 1 u 2 v 2 v 2       u . v = u T v = � �  .  = u 1 u 2 . . . u n . . .       . . .       . . .     u n v n v n Sachin Mehta ( University of Washington, Seattle Email: sacmehta@uw.edu ) CSE 446: Linear Algebra Review 5 / 15

Matrices A matrix A N × M is written as:   a 11 a 12 . . . a 1 M a 21 a 22 . . . a 2 M   A =  . . . .  . . . .   . . . .   a N 1 a N 2 . . . a NM Addition and Subtraction C = A ± B = a ij ± b ij Matrix Product: The product of matrix A n × m and B m × p is another matrix C n × p given by the formula: m � C = AB ⇐ ⇒ c ij = a ik b kj k =1 Note: Matrix multiplication is not commutative ( AB � = BA ). Sachin Mehta ( University of Washington, Seattle Email: sacmehta@uw.edu ) CSE 446: Linear Algebra Review 6 / 15

Matrices Inverse of a matrix Matrices can be divided by scalar, but how can we divide a matrix by a matrix? Take Inverse Inverse of a matrix A is denoted by A − 1 . If A is a square matrix, then AA − 1 = I All matrices are not invertible (refer linear algebra course for more details). In general, Invertible matrices are square. Invertible matrices have LINEARLY INDEPENDENT COLUMNS (i.e. no vector formed by a column of the matrix is a scalar multiple of another) The DETERMINANT of the matrix != 0 This is not a linear algebra class. We’ll not ask you to solve large matrix inverses by hand. Instead use software!!. Linear algebra package for Python: numpy.linalg ( https://docs. scipy.org/doc/numpy/reference/routines.linalg.html ) Sachin Mehta ( University of Washington, Seattle Email: sacmehta@uw.edu ) CSE 446: Linear Algebra Review 7 / 15

Matrices Trace of a matrix Tr ( A ) is the sum of diagonal elements of the matrix. n � Tr ( A ) = a ii i =1 Sachin Mehta ( University of Washington, Seattle Email: sacmehta@uw.edu ) CSE 446: Linear Algebra Review 8 / 15

Matrix-Vector Product Multiplying a vector v by a matrix A transforms the vector v into new vector w . w is not always the same dimension as that of v   � v 1   a 11 a 12 a 11 v 1 + a 12 v 2 � w = Av = a 21 a 22 = a 21 v 1 + a 22 v 2     v 2 a 31 v 1 + a 32 v 2 a 31 a 32 Example: rotating a vector by 30 o . Sachin Mehta ( University of Washington, Seattle Email: sacmehta@uw.edu ) CSE 446: Linear Algebra Review 9 / 15

Matrix Differentiation Differentiation of a matrix with respect to scalar function Very useful in Machine Learning - find gradients, find maximums / minimums for optimization Not as different from regular calculus as you may think If entry a ij of matrix A is some function of f ( x ), then the result is a matrix of the form: δ a 11 δ a 12 δ a 1 M   . . . δ x δ x δ x δ a 21 δ a 22 δ a 2 M . . . δ A   δ x δ x δ x δ x =  . . . .  . . . .   . . . .   δ a N 1 δ a N 2 δ a NM . . . δ x δ x δ x Example: � x � 1 x 2 � δ A 2 x � A = δ x = , then 1 x 0 1 Sachin Mehta ( University of Washington, Seattle Email: sacmehta@uw.edu ) CSE 446: Linear Algebra Review 10 / 15

Matrix Differentiation Differentiation of a scalar function with respect to Matrix Also known as Gradient matrix Given a scalar function of a matrix y = f ( X ), the derivative δ y δ X is: δ y δ y δ y   . . . δ x 11 δ x 12 δ x 1 M δ y δ y δ y . . .   δ y δ x 21 δ x 22 δ x 2 M   δ X = . . . .   . . . . . . . .     δ y δ y δ y . . . δ x N 1 δ x N 2 δ x NM Example: Linear Regression N � ( x i . w − y i ) 2 w = arg min w ˆ i =1 How to find arg min? Derivative, of course...but with respect to w , which is a weight vector, not a single variable. Sachin Mehta ( University of Washington, Seattle Email: sacmehta@uw.edu ) CSE 446: Linear Algebra Review 11 / 15

Matrix Differentiation Differentiation of a scalar function with respect to Matrix � N δ i =1 ( x i . w − y i ) 2   N δ w 1 δ . ( x i . w − y i ) 2 = � .   . δ w   i =1 δ � N i =1 ( x i . w − y i ) 2 δ w n � N δ i =1 ( x i 1 w 1 + . . . + x in w n − y i ) 2   δ w 1 . . =   .   δ � N i =1 ( x i 1 w 1 + . . . + x in w n − y i ) 2 δ w n δ � N  i =1 2 x i 1 ( x i . w − y i )  δ w 1 . . =   .   � N δ i =1 2 x in ( x i . w − y i ) δ w n Sachin Mehta ( University of Washington, Seattle Email: sacmehta@uw.edu ) CSE 446: Linear Algebra Review 12 / 15

Eigen Vectors and Eigen Values When a vector x is multiplied by a matrix A (so resultant vector is Ax ), then direction of the resulting vector is changed. For example, rotating a vector. There are certain vectors ( Ax whose direction is the same as x . Such vectors are called eigen vectors. When we multiply A with x , then the resultant vector is scaled by λ . This λ is called an eigen value and helps in determining whether the vector x is stretched or shrunk or reversed or left unchanged when multiplied with A . Matrix A has eigenvector x and eigen value λ if for some x , we have Ax = λ x ( A − λ I ) x = 0 where n solution λ ’s are given by characteristic equation: det ( A − λ I ) = 0. Determinants are tedious to compute by hand. Use mathematical libraries to compute the determinant. Sachin Mehta ( University of Washington, Seattle Email: sacmehta@uw.edu ) CSE 446: Linear Algebra Review 13 / 15

Example Prove that Tr ( AC ) = Tr ( CA ) n � Tr ( AC ) = ( AC ) ii i =1 n n � � = a ij c ji i =1 j =1 n n � � = c ji a ij j =1 i =1 n � = ( CA ) jj j =1 = Tr ( CA ) Sachin Mehta ( University of Washington, Seattle Email: sacmehta@uw.edu ) CSE 446: Linear Algebra Review 14 / 15

Useful Python Libraries/Functions Linear algebra library (numpy.linalg) - This library has functions that are useful for programming assignments Matrix inverse - numpy.linalg.inv Eigen Values - numpy.linalg.eig Dot product of two arrays - numpy.dot Matrix multiplication - numpy.matmul Dot product of two vectors - numpy.vdot Link: https://docs.scipy.org/doc/numpy/reference/ routines.linalg.html Sachin Mehta ( University of Washington, Seattle Email: sacmehta@uw.edu ) CSE 446: Linear Algebra Review 15 / 15