Singular Value Decomposition and Digital Image Compression Chris - PowerPoint PPT Presentation

Singular Value Decomposition and Digital Image Compression Chris Bingham December 12, 2016 Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 1 / 32

Plan What is SVD? Process of SVD Application to Digital Image Compression Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 2 / 32

What is Singular Value Decomposition? Method used to diagonalize a non-square matrix A . Find the singular values of matrix A Use the singular values to find an approximation of matrix A that has a lesser rank that the original matrix. Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 3 / 32

Theorem of Singular Value Decomposition Let A be an m × n matrix with rank r . Then there exists an m × n matrix Σ for which the diagonal entries in D are the first r singular values of A , σ 1 ≥ σ 2 ≥ ... ≥ σ r > 0, and there exist an m × m orthogonal matrix U and an n × n orthogonal matrix V such that V T A = U Σ m × n m × m m × n n × n Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 4 / 32

Example Using a 3x2 Matrix Given the 3x2 matrix below, compute the SVD.   1 1 A = 1 0   0 1 Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 5 / 32

Calculate A T A �   1 1 � 1 � 2 � 1 0 1  = A T A = 1 0  1 0 1 1 2 0 1 By calculating A T A we obtain a square symmetric matrix through which we can find the eigenvalues. Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 6 / 32

Find the Eigenvalues of A T A p ( λ ) = λ 2 − T λ + D p ( λ ) = λ 2 − 4 λ + 3 p ( λ ) =( λ − 3)( λ − 1) We now have the eigenvalues λ 1 = 3 and λ 2 = 1 Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 7 / 32

Construct V � 1 � λ 1 = 3 v 1 = 1 � − 1 � λ 2 = 1 v 2 = 1 Convert v 1 and v 2 to unit vectors and put as columns of matrix V . � 1 − 1 � √ √ 2 2 V = 1 1 √ √ 2 2 Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 8 / 32

Construct Σ The format of Σ is � D � 0 Σ = 0 0 Where � √ � 3 0 D = 0 1 We want Σ to be the same dimesions as the original matrix A . Therefore Σ will be a 3 × 2 matrix. √   3 0 Σ = 0 1   0 0 Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 9 / 32

Construct U Using u k = 1 ( Av k ) σ k Letting k range from 1 to the rank of A , construct the columns of U. Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 10 / 32

Construct U   � 2 � 1    � 1 1 3 1 √  =   2 1 u 1 = √ 1 0    1   √ 3 6   √ 0 1 2 1 √ 6   0    � 1 1 � − 1 u 2 = 1 − 1 √  = 2 1 0   1 √    2 1   1 0 1 √ 2 √ 2 Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 11 / 32

Construct U   � 2 0 3   1 − 1 U =   √ √ 6 2   1 1 √ √ 6 2 Problem! This should be a square matrix by the SVD theorem. To find the third vector use the Gram-Schmidt Process. Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 12 / 32

Using the Gram-Schmidt Process Start by choosing a vector y that is not in the plane of the the two vectors already found for the matrix U .   1 y = 0   0 We can now use this formula from the Gram-Schmidt process to find our third unit vector for U . u 3 = y − y . u 1 u 1 − y . u 2 u 2 u 1 . u 1 u 2 . u 2 Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 13 / 32

Completing Matrix U By the Gram-Schmidt process we obtain our third vector, all we need to do is make it a unit vector. 1   1   √ 3 3 u 3 − 1 − 1 u 3 = || u 3 || =     √ 3 3   − 1 − 1 3 √ 3 We now have the correct matrix U and we can complete the SVD of matrix A .   � 2 1 0 3 √ 3   1 − 1 − 1 U =   √ √ √ 6 2 3   1 1 − 1 √ √ √ 6 2 3 Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 14 / 32

A = U Σ V T V T = Σ A U m × n m × m m × n n × n √   � 2 1 � 1 0     1 1 3 0 � T − 1 3 √ 3  = √ √  1 − 1 − 1  2 2 1 0 0 1      1 1 √ √ √ 6 2 3   √ √ 0 1 0 0 2 2 1 1 − 1 √ √ √ 6 2 3 Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 15 / 32

Check that A = U Σ V T To be able to check that our original matrix is achieved, start by multiplying the matrix U by each column of Σ  v T  1 . .   σ 1 0 0   .   ...  v T   0 0 0   r  � � A = u 1 u 2 · · · u r · · · u m    v T    0 0 σ r   r +1   .   . 0 0   .   v T n  v T  1 . .   .    v T    � � r A = σ 1 u 1 σ 2 u 2 · · · σ r u r 0 · · · 0  v T    r +1  .  .   .   v T n Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 16 / 32

Check that A = U Σ V T Partition the matrices and perform column-row multiplication v T   1 . .  .     v T   r  � � A = σ 1 u 1 σ 2 u 2 · · · σ r u r 0 · · · 0   v T   r +1  .  .   .   v T n A = σ 1 u 1 v T 1 + σ 2 u 2 v T 2 + ... + σ r u r v T r + 0 v T r +1 + 0 v T n A = σ 1 u 1 v T 1 + σ 2 u 2 v T 2 + ... + σ r u r v T r k � σ i u i v T A k = i i =1 Letting k be the desired rank approximation of A Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 17 / 32

Applied to Our 3 × 2 Example Matrix A Our calculated SVD √   � 2 1 � 1 0     1 1 3 0 � T − 1 3 √ 3  = √ √  1 − 1 − 1  2 2 A = 1 0 0 1   1 1    √ √ √ 6 2 3   0 1 0 0 √ √ 1 1 − 1 2 2 √ √ √ 6 2 3 Which has the form   σ 1 0 � v T � � � 1 A = 0 u 1 u 2 u 3 σ 2   v T 2 0 0 A = σ 1 u 1 v T 1 + σ 2 u 2 v T 2 Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 18 / 32

Applied to Our 3 × 2 Example Matrix A A = σ 1 u 1 v T 1 + σ 2 u 2 v T 2   �   2 0 √ 3 � � � � − 1 1 1 − 1 1   1 A = 3 + 1   √   √ √ 2 √ √ 2 2 2 2 √ 6   1   1 √ 2 √ 6 1 1     0 0 √ √ √ 3 3 1 1 1 − 1 A = 3  + 1   √ √  2 2  12 12  − 1 1 1 1 2 2 √ √ 12 12     1 1 0 0  + 1 1 1 − 1 A =    2 2 2 2 1 1 − 1 1 2 2 2 2 Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 19 / 32

Application to Digital Image Compression Digital images are stored as pixels, each pixel representing a cell in a matrix. Each cell represents the intensity of the pixel and has an associated value. Pixel data can be extracted from an image and processed through SVD to gain an compressed version of the image Visual representation of SVD can be seen by displaying the approximated pixel data using varying amounts of singular values. Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 20 / 32

Original Image Original image that data was extracted from to create matrix A . Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 21 / 32

Using 1 Singular Value The image is unrecognizable. While the file size is tiny, this is not a useful compression of the original image. Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 22 / 32

Using 2 Singular Values Just one rank added to the approximation made a noticable difference. As the rank gets larger the difference will be indistinguishable. Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 23 / 32

Using 5 Singular Values Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 24 / 32

Using 10 Singular Values Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 25 / 32

Using 20 Singular Values Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 26 / 32

Using 40 Singular Values Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 27 / 32

Using 80 Singular Values Chris Bingham Singular Value Decompositionand Digital Image Compression December 12, 2016 28 / 32