SI231 Matrix Computations Lecture 0: Overview Ziping Zhao Fall Term 2020–2021 School of Information Science and Technology ShanghaiTech University, Shanghai, China
Course Information Ziping Zhao 1
General Information • Instructor: Prof. Ziping Zhao – office: Rm. 1C-503C, SIST Building – e-mail: zhaoziping@shanghaitech.edu.cn – website: http://faculty.sist.shanghaitech.edu.cn/faculty/zhaoziping • Lecture hours and venue: – Tuesday/Thursday 10:15am–11:55am, Rm. 101, Teaching Center • Course helpers whom you can consult: – Lin Zhu, zhulin@shanghaitech.edu.cn (leading TA) – Zhihang Xu, xuzhh@... ; Jiayi Chang, changyj@... – Song Mao, maosong@... ; Zhicheng Wang, wangzhch1@... – Sihang Xu, xush@... ; Xinyue Zhang, zhangxy11@... – Chenguang Zhang, zhangchg@... ; Bing Jiang, jiangbing@... • Course website: http://faculty.sist.shanghaitech.edu.cn/faculty/ zhaoziping/si231 Ziping Zhao 2
Course Contents • This is a foundation course on matrix analysis and computations, which are widely used in many different fields, e.g., – machine learning, computer vision and graphics, natural language processing, – systems and control, signal and image processing, communications, networks, – optimization, statistics, econometrics, finance, and many more... • Aim: covers matrix analysis and computations at an advanced or research level. • Scope: – basic matrix concepts, subspace, norms, – linear system of equations, LU decomposition, Cholesky decomposition – linear least squares – eigendecomposition, singular value decomposition – positive semidefinite matrices – pseudo-inverse, QR decomposition – (advanced) tensor decomposition, advanced matrix calculus, compressive sens- ing, structured matrix factorization Ziping Zhao 3
Learning Resources • Textbook: – Gene H. Golub and Charles F. van Loan, Matrix Computations (Fourth Edition), The John Hopkins University Press, 2013. • Recommended readings: – Roger A. Horn and Charles R. Johnson, Matrix Analysis (Second Edition), Cambridge University Press, 2012. – Jan R. Magnus and Heinz Neudecker, Matrix Differential Calculus with Appli- cations in Statistics and Econometrics (Third Edition), John Wiley and Sons, New York, 2007. – Gilbert Strang, Linear Algebra and Learning from Data , Wellesley-Cambridge Press, 2019. – Giuseppe Calafiore and Laurent El Ghaoui, Optimization Models, Cambridge University Press, 2014. Ziping Zhao 4
Assessment and Academic Honesty • Assessment: – Assignments: 30% ∗ may contain MATLAB questions ∗ where to submit: ShanghaiTech e-learning platform, i.e., Blackboard (Bb) ∗ no late submissions would be accepted, except for exceptional cases. – Mid-term examination: 40% – Final project: 30% • Academic honesty: – Students are strongly advised to read the ShanghaiTech Policy on Academic Integrity: https://oaa.shanghaitech.edu.cn/2015/0706/c4076a31250/ page.htm Ziping Zhao 5
Additional Notice • Sitting in is welcome, and please send the Leading TA your e-mail address to keep you updated with the course. • You can also get consultation from me; send me an email for an appointment • Do regularly check your ShanghaiTech e-mail address; this is the only way we can reach you • The e-learning platform Blackboard (Bb) will be used to announce scores and for online homework submission Ziping Zhao 6
A Glimpse of Topics Ziping Zhao 7
Linear System of Equations • Problem: given A ∈ R n × n , y ∈ R n , solve Ax = y . • Question 1: How to solve it? – don’t tell me answers like x=inv(A)*y or x=A \ y on MATLAB! – this is about matrix computations • Question 2: How to solve it when n is very large? – it’s too slow to do the generic trick x=A \ y when n is very large – getting better understanding of matrix computations will enable you to exploit problem structures to build efficient solvers • Question 3: How sensitive is the solution x when A and y contain errors? – key to system analysis, or building robust solutions Ziping Zhao 8
Application Example: Electrical Circuit • In a given circuit if enough values of currents, resistance, and potential difference is known, we should be able to find the other unknown values of these quantities. • Mainly use Ohm’s Law, Kirchhoff’s Voltage Law, and Kirchhoff’s Current Law. y A x � �� � � �� � � �� � 1 1 − 1 0 I 1 = . − R 1 − R 2 − R 3 0 E 2 + E 3 R 5 I 2 − R 4 0 − R 5 − E 3 − E 1 I 3 • Imagine we have a much more complicated circuit network... Ziping Zhao 9
Least Squares (LS) • Problem: given A ∈ R m × n , y ∈ R n , solve x ∈ R n � y − Ax � 2 min 2 , �� n i =1 | x i | 2 . where � · � 2 is the Euclidean norm; i.e., � x � 2 = • widely used in science, engineering, and mathematics • assuming a tall and full-rank A , the LS solution is uniquely given by x LS = ( A T A ) − 1 A T y . Ziping Zhao 10
Application Example: Linear Prediction (LP) • let { y t } t ≥ 0 be a time series. • Model (autoregressive (AR) model): y t = a 1 y t − 1 + a 2 y t − 2 + · · · + a q y t − q + v t , t = 0 , 1 , 2 , . . . for some coefficients { a i } q i =1 , where v t is noise or modeling error. • Problem: estimate { a i } q i =1 from { y t } t ≥ 0 ; can be formulated as LS • Applications: time-series prediction, speech analysis and coding, spectral estimation . . . Ziping Zhao 11
A Toy Demo: Predicting Hang Seng Index 4 2.3 x 10 Hang Seng Index Linear Prediction 2.25 2.2 Hang Seng Index 2.15 2.1 2.05 2 1.95 1.9 0 10 20 30 40 50 60 day blue— Hang Seng Index during a certain time period. red— training phase; the line is � q i =1 a i y t − i ; a is obtained by LS; q = 10 . y t = � q green— prediction phase; the line is ˆ i =1 a i ˆ y t − i ; the same a in the training phase. Ziping Zhao 12
A Real Example: Real-Time Prediction of Flu Activity Tracking influenza outbreaks by ARGO — a model combining the AR model and GOogle search data. Source: [Yang-Santillana-Kou2015] . Ziping Zhao 13
Eigenvalue Problem • Problem: given A ∈ R n × n , find a v ∈ R n such that Av = λ v , for some λ . • Eigendecomposition: let A ∈ R n × n be symmetric; i.e., a ij = a ji for all i, j . It also admits a decomposition/factorization A = VΛV T , where V ∈ R n × n is orthogonal, i.e., V T V = I ; Λ = Diag( λ 1 , . . . , λ n ) • also widely used, either as an analysis tool or as a computational tool • no closed form in general; can be numerically computed Ziping Zhao 14
Application Example: PageRank • PageRank is an algorithm used by Google to rank the pages of a search result. • the idea is to use counts of links of various pages to determine pages’ importance. Source: Wiki. Ziping Zhao 15
One-Page Explanation of How PageRank Works • Model: v j � = v i , i = 1 , . . . , n, c j j ∈L i where c j is the number of outgoing links from page j ; L i is the set of pages with a link to page i ; v i is the importance score of page i . • as an example, A v v � �� � � �� � � �� � 1 1 0 1 v 1 v 1 2 3 1 0 0 0 v 2 v 2 3 = . 1 1 0 0 v 3 v 3 2 3 0 0 0 0 v 4 v 4 • finding v is an eigenvalue problem—with n being of order of millions! • further reading: [Bryan-Tanya2006] Ziping Zhao 16
Low-Rank Matrix Approximation given Y ∈ R m × n and an integer r < min { m, n } , find an ( A , B ) ∈ • Problem: R m × r × R r × n such that either Y = AB or Y ≈ AB . … • Formulation: A ∈ R m × r , B ∈ R r × n � Y − AB � 2 min F , where � · � F is the Frobenius, or matrix Euclidean, norm. • Applications: dimensionality reduction, extracting meaningful features from data, low-rank modeling, . . . Ziping Zhao 17
Singular Value Decomposition (SVD) • SVD: Any Y ∈ R m × n can be decomposed/factorized into Y = UΣV T , where U ∈ R m × m , V ∈ R n × n are orthogonal; Σ ∈ R m × n takes a diagonal form. • also a widely used analysis and computational tool; can be numerically computed • SVD can be used to solve the low-rank matrix approximation problem A ∈ R m × r , B ∈ R r × n � Y − AB � 2 min F . Ziping Zhao 18
Application Example: Image Compression • let Y ∈ R m × n be an image. original image, size = 101 x 1202 • store the low-rank factor pair ( A , B ) , instead of Y . truncated SVD, r = 3 truncated SVD, r = 5 truncated SVD, r = 10 truncated SVD, r = 20 Ziping Zhao 19
Application Example: Principal Component Analysis (PCA) given a set of data points { y 1 , y 2 , . . . , y n } ⊂ R m and an integer • Aim: r < min { m, n } , perform a low-dimensional representation y i = Qc i + µ + e i , i = 1 , . . . , n, where Q ∈ R m × r is a basis; c i ’s are coefficients; µ is a base; e i ’s are errors Ziping Zhao 20
Toy Demo: Dimensionality Reduction of a Face Image Dataset A face image dataset. Image size = 112 × 92 , number of face images = 400 . Each y i is the vectorization of one face image, leading to m = 112 × 92 = 10304 , n = 400 . Ziping Zhao 21
Recommend
More recommend
