Partial Lanczos SVD methods for R Bryan Lewis 1 , adapted from the - PowerPoint PPT Presentation

Partial Lanczos SVD methods for R Bryan Lewis 1 , adapted from the work of Jim Baglama 2 and Lothar Reichel 3 1 REvolution Computing 2 University of Rhode Island 3 Kent State University UseR2009 Partial Lanczos SVD methods for R

Outline SVD and partial SVD Partial Lanczos bidiagonalization The irlba package Partial Lanczos SVD methods for R

SVD Let A ∈ R ℓ × n , ℓ ≥ n . n � σ j u j v T A = j , j = 1 � 1 if j = k , v T j v k = u T j u k = 0 o.w., u j ∈ R ℓ , v j ∈ R n , j = 1 , 2 , . . . , n , and σ 1 ≥ σ 2 ≥ · · · ≥ σ n ≥ 0 . Partial Lanczos SVD methods for R

Partial SVD Let k < n . k ˜ � σ j u j v T A k := j j = 1 Partial Lanczos SVD methods for R

Partial Lanczos bi-diagonalization Start with a given vector p 1 . Compute m steps of the Lanczos process: AP m = Q m B m A T Q m P m B T m + r m e T = m , B m ∈ R m × m , P m ∈ R n × m , Q m ∈ R ℓ × m , P T m P m = Q T m Q m = I m , r m ∈ R n , P T m r m = 0 , P m = [ p 1 , p 2 , . . . , p m ] . Partial Lanczos SVD methods for R

Approximating partial SVD with partial Lanczos bi-diagonalization A T AP m A T Q m B m = P m B T m B m + r m e T = m B m , Partial Lanczos SVD methods for R

Approximating partial SVD with partial Lanczos bi-diagonalization A T AP m A T Q m B m = P m B T m B m + r m e T = m B m , AA T Q m AP m B T m + Ar m e T = m , Q m B m B T m + Ar m e T = m . Partial Lanczos SVD methods for R

Approximating partial SVD with partial Lanczos bi-diagonalization Let m � T � � σ B j u B v B B m = . j j j = 1 Partial Lanczos SVD methods for R

Approximating partial SVD with partial Lanczos bi-diagonalization Let m � T � � σ B j u B v B B m = . j j j = 1 � � i.e., B m v B j = σ B j u B j , and B T m u b j = σ B j v B j . Define: Partial Lanczos SVD methods for R

Approximating partial SVD with partial Lanczos bi-diagonalization Let m � T � � σ B j u B v B B m = . j j j = 1 � � i.e., B m v B j = σ B j u B j , and B T m u b j = σ B j v B j . Define: σ B σ j ˜ := j , Q m u B ˜ u j := j , P m v B ˜ v j := j . Partial Lanczos SVD methods for R

Partial SVD approximation of A AP m v B A ˜ v j = j Q m B m v B = j σ B j Q m u B = j σ j ˜ = ˜ u j , Partial Lanczos SVD methods for R

Partial SVD approximation of A AP m v B A ˜ v j = j Q m B m v B = j σ B j Q m u B = j σ j ˜ = ˜ u j , A T ˜ A T Q m u B u j = j P m B T m u B j + r m e T m u B = j σ B j P m v B j + r m e T m u B = j v j + r m e T m u B σ j ˜ = ˜ j . Partial Lanczos SVD methods for R

Augment and restart 1. Compute the Lanczos process up to step m . 2. Compute k < m approximate singular vectors. 3. Orthogonalize against the approximate singular vectors to get a new starting vector. 4. Continue the Lanczos process with the new starting vector for m more steps. 5. Check for convergence tolerance and exit if met. 6. GOTO 1. Partial Lanczos SVD methods for R

Sketch of the augmented process... ¯ [˜ v 1 , ˜ v 2 , . . . , ˜ P k + 1 := v k , p m + 1 ] , Partial Lanczos SVD methods for R

Sketch of the augmented process... ¯ [˜ v 1 , ˜ v 2 , . . . , ˜ P k + 1 := v k , p m + 1 ] , A ¯ σ 1 ˜ σ 1 ˜ σ k ˜ P k + 1 = [˜ u 1 , ˜ u 2 , . . . , ˜ u k , Ap m + 1 ] Partial Lanczos SVD methods for R

Sketch of the augmented process... ¯ [˜ v 1 , ˜ v 2 , . . . , ˜ P k + 1 := v k , p m + 1 ] , A ¯ σ 1 ˜ σ 1 ˜ σ k ˜ P k + 1 = [˜ u 1 , ˜ u 2 , . . . , ˜ u k , Ap m + 1 ] Ap m + 1 = � k u j } k Orthogonalize Ap m + 1 against { ˜ j = 1 ρ j ˜ j = 1 : u j + r k . Partial Lanczos SVD methods for R

Sketch of the augmented process... ¯ [˜ v 1 , ˜ v 2 , . . . , ˜ P k + 1 := v k , p m + 1 ] , A ¯ σ 1 ˜ σ 1 ˜ σ k ˜ P k + 1 = [˜ u 1 , ˜ u 2 , . . . , ˜ u k , Ap m + 1 ] Ap m + 1 = � k u j } k Orthogonalize Ap m + 1 against { ˜ j = 1 ρ j ˜ j = 1 : u j + r k . ¯ [˜ u 1 , ˜ u 2 , . . . , ˜ Q k + 1 := u k , r k / � r k � ] ,   σ 1 ˜ ρ 1 σ 2 ˜ ρ 2   ¯ B k + 1 :=  .   ...   ρ k  � r k � Partial Lanczos SVD methods for R

Sketch of the augmented process... ¯ [˜ v 1 , ˜ v 2 , . . . , ˜ P k + 1 := v k , p m + 1 ] , A ¯ σ 1 ˜ σ 1 ˜ σ k ˜ P k + 1 = [˜ u 1 , ˜ u 2 , . . . , ˜ u k , Ap m + 1 ] Ap m + 1 = � k u j } k Orthogonalize Ap m + 1 against { ˜ j = 1 ρ j ˜ j = 1 : u j + r k . ¯ [˜ u 1 , ˜ u 2 , . . . , ˜ Q k + 1 := u k , r k / � r k � ] ,   σ 1 ˜ ρ 1 σ 2 ˜ ρ 2   ¯ B k + 1 :=  .   ...   ρ k  � r k � A ¯ P k + 1 = ¯ Q k + 1 ¯ B k + 1 . Partial Lanczos SVD methods for R

The irlba package Usage: irlba (A, nu = 5, nv = 5, adjust = 3, aug = "ritz", sigma = "ls", maxit = 1000, reorth = 1, tol = 1e-06, V = NULL) Partial Lanczos SVD methods for R

Small example > A<-matrix (rnorm(5000*5000),5000,5000) > require (irlba) > system.time (L<-irlba (A,nu=5,nv=5)) user system elapsed 41.640 0.470 36.985 > gc() used (Mb) ... max used (Mb) Ncells 143301 7.7 ... 350000 18.7 Vcells 25193378 192.3 ... 78709588 600.6 Partial Lanczos SVD methods for R

Small example (continued) > system.time (S<-svd(A,nu=5,nv=5)) user system elapsed 616.035 4.396 187.371 > gc() used (Mb) ... max used (Mb) Ncells 143312 7.7 ... 144539 7.8 Vcells 25248388 192.7 ... 200285943 1528.1 Partial Lanczos SVD methods for R

Small example (continued) > system.time (S<-svd(A,nu=5,nv=5)) user system elapsed 616.035 4.396 187.371 > gc() used (Mb) ... max used (Mb) Ncells 143312 7.7 ... 144539 7.8 Vcells 25248388 192.7 ... 200285943 1528.1 > sqrt (crossprod(S$d[1:5]-L$d)/crossprod(S$d[1:5])) [,1] [1,] 1.56029e-12 Partial Lanczos SVD methods for R

Large examples (live demo) The R implementation of IRLBA supports: ◮ Dense real/complex in-process matrices (normal R matrices) ◮ Sparse real in-process matrices (Matrix) ◮ Dense, real in- or out-of-process huge matrices with bigmemory + bigalgebra Partial Lanczos SVD methods for R

References 1. http: // www.rforge.net / irlba 2. http: // www.math.uri.edu / ∼ jbaglama 3. http: // www.math.kent.edu / ∼ reichel 4. http: // www.math.kent.edu / ∼ blewis 5. http: // revolution-computing.com Partial Lanczos SVD methods for R