Efficient Algorithms and Error Analysis for Motivation the Modified - PowerPoint PPT Presentation

The Modified Nyström Method Wang & Zhang Efficient Algorithms and Error Analysis for Motivation the Modified Nyström Method The Nyström Method Column Sampling Shusen Wang 1 Zhihua Zhang 2 Improve the Nyström Method 1 Zhejiang University, China The Modified Nyström 2 Shanghai Jiao Tong University, China Method Comparisons between the Two Methods AISTATS 2014 Efficient Algorithms Theories

Outline The Modified Nyström Method 1 Motivation Wang & Zhang The Nyström Method 2 Motivation The Nyström Method Column Sampling 3 Column Sampling Improve the Improve the Nyström Method 4 Nyström Method The Modified The Modified Nyström Method 5 Nyström Method Comparisons between the Two Methods Comparisons between the Two Efficient Algorithms Methods Efficient Algorithms Theories Theories

Outline The Modified Nyström Method 1 Motivation Wang & Zhang The Nyström Method 2 Motivation The Nyström Method Column Sampling 3 Column Sampling Improve the Improve the Nyström Method 4 Nyström Method The Modified The Modified Nyström Method 5 Nyström Method Comparisons between the Two Methods Comparisons between the Two Efficient Algorithms Methods Efficient Algorithms Theories Theories

Kernel methods The Modified Nyström Method Wang & Zhang K : n × n kernel matrix. Motivation Matrix inverse b = ( K + α I n ) − 1 y The Nyström time complexity: O ( n 3 ) Method performed by Gaussian process regression, least Column Sampling square SVM, kernel ridge regression Improve the Partial eigenvalue decomposition of K Nyström Method time complexity: O ( n 2 k ) The Modified Nyström performed by kernel PCA and some manifold learning Method methods Comparisons between the Two Methods Efficient Algorithms Theories

Kernel methods The Modified Nyström Method Wang & Zhang K : n × n kernel matrix. Motivation Matrix inverse b = ( K + α I n ) − 1 y The Nyström time complexity: O ( n 3 ) Method performed by Gaussian process regression, least Column Sampling square SVM, kernel ridge regression Improve the Partial eigenvalue decomposition of K Nyström Method time complexity: O ( n 2 k ) The Modified Nyström performed by kernel PCA and some manifold learning Method methods Comparisons between the Two Methods Efficient Algorithms Theories

Kernel methods The Modified Nyström Method Wang & Zhang K : n × n kernel matrix. Motivation Matrix inverse b = ( K + α I n ) − 1 y The Nyström time complexity: O ( n 3 ) Method performed by Gaussian process regression, least Column Sampling square SVM, kernel ridge regression Improve the Partial eigenvalue decomposition of K Nyström Method time complexity: O ( n 2 k ) The Modified Nyström performed by kernel PCA and some manifold learning Method methods Comparisons between the Two Methods Efficient Algorithms Theories

Computational Challenges The Modified Nyström Method Wang & Zhang High time complexities: O ( n 3 ) or O ( n 2 k ) Motivation High space complexity: O ( n 2 ) The Nyström Method the iterative algorithms go many passes through the Column data Sampling you had better put the entire kernel matrix in RAM Improve the Nyström if the data does not fit in the RAM Method ⇒ one swap between RAM and disk in each pass The Modified Nyström ⇒ very slow! Method Comparisons between the Two Methods Efficient Algorithms Theories

Computational Challenges The Modified Nyström Method Wang & Zhang High time complexities: O ( n 3 ) or O ( n 2 k ) Motivation High space complexity: O ( n 2 ) The Nyström Method the iterative algorithms go many passes through the Column data Sampling you had better put the entire kernel matrix in RAM Improve the Nyström if the data does not fit in the RAM Method ⇒ one swap between RAM and disk in each pass The Modified Nyström ⇒ very slow! Method Comparisons between the Two Methods Efficient Algorithms Theories

How to Speedup The Modified Nyström Method Wang & Zhang If we can find a fast low-rank factorization Motivation The Nyström D T Method K ≈ D , ���� ���� ���� Column n × n n × d d × n Sampling Improve the then ( K + α I n ) − 1 and the partial eigenvalue Nyström Method decomposition of K can be approximated solved highly The Modified Nyström efficiently. Method Comparisons between the Two Methods Efficient Algorithms Theories

How to Speedup: Example 1 The Modified Nyström Suppose we have a low-rank factorization Method D T Wang & ≈ . K D Zhang ���� ���� ���� n × n n × d d × n Motivation Approximately compute the matrix inverse ( K + α I n ) − 1 The Nyström Method as follows. Column Expand ( DD T + α I n ) − 1 using the Sampling Improve the Sherman-Morrison-Woodbury formula and obtain Nyström Method � − 1 = α − 1 I n − α − 1 D � − 1 D T � DD T + α I n � α I d + D T D . The Modified ���� ���� � �� � Nyström n × d d × n d × d Method Comparisons between the Two It costs only O ( nd 2 ) time and O ( nd ) space to compute Methods Efficient Algorithms Theories � DD T + α I n � − 1 y . b =

How to Speedup: Example 2 The Modified Nyström Suppose we have a low-rank factorization Method D T Wang & K ≈ D , Zhang ���� ���� ���� n × n n × d d × n Motivation Compute the eigenvalue decomposition of K as follows. The Nyström Method Column Sampling Compute the eigenvalue decomposition of the d × d Improve the small matrix S = D T D ∈ R d × d : Nyström Method S = U S Λ S U T S . The Modified Nyström The partial eigenvalue decomposition of DD T is Method Comparisons between the Two Methods K ≈ DD T = � � � � T DU S Λ − 1 / 2 DU S Λ − 1 / 2 Efficient Algorithms Λ S Theories S S It costs only O ( nd 2 ) time and O ( nd ) space.

Outline The Modified Nyström Method 1 Motivation Wang & Zhang The Nyström Method 2 Motivation The Nyström Method Column Sampling 3 Column Sampling Improve the Improve the Nyström Method 4 Nyström Method The Modified The Modified Nyström Method 5 Nyström Method Comparisons between the Two Methods Comparisons between the Two Efficient Algorithms Methods Efficient Algorithms Theories Theories

The Nyström Method The Modified Nyström Method Random Selection : Wang & Zhang selects c ( ≪ n ) columns of K to construct C using some Motivation randomized algorithms. After permutation we have The Nyström � W � W Method � � K T 21 K = , C = . Column K 21 K 21 K 22 Sampling Improve the Nyström Method The Nyström Approximation : ˜ K nys ≈ K c The Modified Nyström Method K nys ˜ W † C T = C . Comparisons c between the Two ���� ���� ���� ���� Methods n × c c × c c × n n × n Efficient Algorithms Theories

The Nyström Method The Modified Nyström Method Random Selection : Wang & Zhang selects c ( ≪ n ) columns of K to construct C using some Motivation randomized algorithms. After permutation we have The Nyström � W � W Method � � K T 21 K = , C = . Column K 21 K 21 K 22 Sampling Improve the Nyström Method The Nyström Approximation : ˜ K nys ≈ K c The Modified Nyström Method K nys ˜ W † C T = C . Comparisons c between the Two ���� ���� ���� ���� Methods n × c c × c c × n n × n Efficient Algorithms Theories

The Nyström Approximation The Modified Nyström The Nyström Approximation : Method Wang & Zhang K nys ˜ CW † C T K ≈ = c Motivation (A low-rank factorization). The Nyström Method Column Sampling Improve the Nyström Method Nyström × × Approximation The Modified Nyström Method c × c c × n Comparisons between the Two Methods Efficient Algorithms n × c n × n Theories

Outline The Modified Nyström Method 1 Motivation Wang & Zhang The Nyström Method 2 Motivation The Nyström Method Column Sampling 3 Column Sampling Improve the Improve the Nyström Method 4 Nyström Method The Modified The Modified Nyström Method 5 Nyström Method Comparisons between the Two Methods Comparisons between the Two Efficient Algorithms Methods Efficient Algorithms Theories Theories

Problem Formulation The Modified Nyström Method Wang & Zhang Problem: How to select informative columns of K ∈ R n × n to Motivation The Nyström construct C ∈ R n × c ? Method � K − CUC T � � Column The approximation error F or � Sampling � � K − CUC T � 2 should be as small as possible. � Improve the Nyström Method Hardness: The Modified Nyström Totally ( n Method c ) choices. Comparisons between the Two Methods Efficient Algorithms Theories

Problem Formulation The Modified Nyström Method Wang & Zhang Problem: How to select informative columns of K ∈ R n × n to Motivation The Nyström construct C ∈ R n × c ? Method � K − CUC T � � Column The approximation error F or � Sampling � � K − CUC T � 2 should be as small as possible. � Improve the Nyström Method Hardness: The Modified Nyström Totally ( n Method c ) choices. Comparisons between the Two Methods Efficient Algorithms Theories