Need for iteration Krylov subspaces David S. Watkins Hessenberg form is not the goal. Triangular form is the goal. Hessenberg is as far as we can get with a direct method. (Abel, Galois) need to iterate
Need for iteration Krylov subspaces David S. Watkins Hessenberg form is not the goal. Triangular form is the goal. Hessenberg is as far as we can get with a direct method. (Abel, Galois) need to iterate
Need for iteration Krylov subspaces David S. Watkins Hessenberg form is not the goal. Triangular form is the goal. Hessenberg is as far as we can get with a direct method. (Abel, Galois) need to iterate
Need for iteration Krylov subspaces David S. Watkins Hessenberg form is not the goal. Triangular form is the goal. Hessenberg is as far as we can get with a direct method. (Abel, Galois) need to iterate
Power method, Subspace iteration Krylov subspaces David S. Watkins v , Av , A 2 v , A 3 v , . . . convergence rate | λ 2 /λ 1 | S , A S , A 2 S , A 3 S , . . . subspaces of dimension j ( | λ j +1 /λ j | )
Power method, Subspace iteration Krylov subspaces David S. Watkins v , Av , A 2 v , A 3 v , . . . convergence rate | λ 2 /λ 1 | S , A S , A 2 S , A 3 S , . . . subspaces of dimension j ( | λ j +1 /λ j | )
Power method, Subspace iteration Krylov subspaces David S. Watkins v , Av , A 2 v , A 3 v , . . . convergence rate | λ 2 /λ 1 | S , A S , A 2 S , A 3 S , . . . subspaces of dimension j ( | λ j +1 /λ j | )
Power method, Subspace iteration Krylov subspaces David S. Watkins v , Av , A 2 v , A 3 v , . . . convergence rate | λ 2 /λ 1 | S , A S , A 2 S , A 3 S , . . . subspaces of dimension j ( | λ j +1 /λ j | )
Power method, Subspace iteration Krylov subspaces David S. Watkins v , Av , A 2 v , A 3 v , . . . convergence rate | λ 2 /λ 1 | S , A S , A 2 S , A 3 S , . . . subspaces of dimension j ( | λ j +1 /λ j | )
Power method, Subspace iteration Krylov subspaces David S. Watkins v , Av , A 2 v , A 3 v , . . . convergence rate | λ 2 /λ 1 | S , A S , A 2 S , A 3 S , . . . subspaces of dimension j ( | λ j +1 /λ j | )
Power method, Subspace iteration Krylov subspaces David S. Watkins for greater flexibility shifts ρ 1 , . . . , ρ m ( m small) (explanation deferred) p ( A ) = ( A − ρ 1 I ) · · · ( A − ρ m I ) Substitute p ( A ) for A S , p ( A ) S , p ( A ) 2 S , p ( A ) 3 S , . . . convergence rate | p ( λ j +1 ) / p ( λ j ) |
Power method, Subspace iteration Krylov subspaces David S. Watkins for greater flexibility shifts ρ 1 , . . . , ρ m ( m small) (explanation deferred) p ( A ) = ( A − ρ 1 I ) · · · ( A − ρ m I ) Substitute p ( A ) for A S , p ( A ) S , p ( A ) 2 S , p ( A ) 3 S , . . . convergence rate | p ( λ j +1 ) / p ( λ j ) |
Power method, Subspace iteration Krylov subspaces David S. Watkins for greater flexibility shifts ρ 1 , . . . , ρ m ( m small) (explanation deferred) p ( A ) = ( A − ρ 1 I ) · · · ( A − ρ m I ) Substitute p ( A ) for A S , p ( A ) S , p ( A ) 2 S , p ( A ) 3 S , . . . convergence rate | p ( λ j +1 ) / p ( λ j ) |
Power method, Subspace iteration Krylov subspaces David S. Watkins for greater flexibility shifts ρ 1 , . . . , ρ m ( m small) (explanation deferred) p ( A ) = ( A − ρ 1 I ) · · · ( A − ρ m I ) Substitute p ( A ) for A S , p ( A ) S , p ( A ) 2 S , p ( A ) 3 S , . . . convergence rate | p ( λ j +1 ) / p ( λ j ) |
Power method, Subspace iteration Krylov subspaces David S. Watkins for greater flexibility shifts ρ 1 , . . . , ρ m ( m small) (explanation deferred) p ( A ) = ( A − ρ 1 I ) · · · ( A − ρ m I ) Substitute p ( A ) for A S , p ( A ) S , p ( A ) 2 S , p ( A ) 3 S , . . . convergence rate | p ( λ j +1 ) / p ( λ j ) |
Power method, Subspace iteration Krylov subspaces David S. Watkins for greater flexibility shifts ρ 1 , . . . , ρ m ( m small) (explanation deferred) p ( A ) = ( A − ρ 1 I ) · · · ( A − ρ m I ) Substitute p ( A ) for A S , p ( A ) S , p ( A ) 2 S , p ( A ) 3 S , . . . convergence rate | p ( λ j +1 ) / p ( λ j ) |
Power method, Subspace iteration Krylov subspaces David S. Watkins for greater flexibility shifts ρ 1 , . . . , ρ m ( m small) (explanation deferred) p ( A ) = ( A − ρ 1 I ) · · · ( A − ρ m I ) Substitute p ( A ) for A S , p ( A ) S , p ( A ) 2 S , p ( A ) 3 S , . . . convergence rate | p ( λ j +1 ) / p ( λ j ) |
Subspace Iteration with changes of coordinate system Krylov subspaces David S. take S = span { e 1 , . . . , e j } Watkins p ( A ) S = span { p ( A ) e 1 , . . . , p ( A ) e j } = span { q 1 , . . . , q j } (orthonormal?) build unitary (or not) Q = [ q 1 · · · q j · · · ] change coordinate system: ˆ A = Q − 1 AQ q k → Q − 1 q k = e k span { q 1 , . . . , q j } → span { e 1 , . . . , e j } ready for next iteration
Subspace Iteration with changes of coordinate system Krylov subspaces David S. take S = span { e 1 , . . . , e j } Watkins p ( A ) S = span { p ( A ) e 1 , . . . , p ( A ) e j } = span { q 1 , . . . , q j } (orthonormal?) build unitary (or not) Q = [ q 1 · · · q j · · · ] change coordinate system: ˆ A = Q − 1 AQ q k → Q − 1 q k = e k span { q 1 , . . . , q j } → span { e 1 , . . . , e j } ready for next iteration
Subspace Iteration with changes of coordinate system Krylov subspaces David S. take S = span { e 1 , . . . , e j } Watkins p ( A ) S = span { p ( A ) e 1 , . . . , p ( A ) e j } = span { q 1 , . . . , q j } (orthonormal?) build unitary (or not) Q = [ q 1 · · · q j · · · ] change coordinate system: ˆ A = Q − 1 AQ q k → Q − 1 q k = e k span { q 1 , . . . , q j } → span { e 1 , . . . , e j } ready for next iteration
Subspace Iteration with changes of coordinate system Krylov subspaces David S. take S = span { e 1 , . . . , e j } Watkins p ( A ) S = span { p ( A ) e 1 , . . . , p ( A ) e j } = span { q 1 , . . . , q j } (orthonormal?) build unitary (or not) Q = [ q 1 · · · q j · · · ] change coordinate system: ˆ A = Q − 1 AQ q k → Q − 1 q k = e k span { q 1 , . . . , q j } → span { e 1 , . . . , e j } ready for next iteration
Subspace Iteration with changes of coordinate system Krylov subspaces David S. take S = span { e 1 , . . . , e j } Watkins p ( A ) S = span { p ( A ) e 1 , . . . , p ( A ) e j } = span { q 1 , . . . , q j } (orthonormal?) build unitary (or not) Q = [ q 1 · · · q j · · · ] change coordinate system: ˆ A = Q − 1 AQ q k → Q − 1 q k = e k span { q 1 , . . . , q j } → span { e 1 , . . . , e j } ready for next iteration
Subspace Iteration with changes of coordinate system Krylov subspaces David S. take S = span { e 1 , . . . , e j } Watkins p ( A ) S = span { p ( A ) e 1 , . . . , p ( A ) e j } = span { q 1 , . . . , q j } (orthonormal?) build unitary (or not) Q = [ q 1 · · · q j · · · ] change coordinate system: ˆ A = Q − 1 AQ q k → Q − 1 q k = e k span { q 1 , . . . , q j } → span { e 1 , . . . , e j } ready for next iteration
Subspace Iteration with changes of coordinate system Krylov subspaces David S. take S = span { e 1 , . . . , e j } Watkins p ( A ) S = span { p ( A ) e 1 , . . . , p ( A ) e j } = span { q 1 , . . . , q j } (orthonormal?) build unitary (or not) Q = [ q 1 · · · q j · · · ] change coordinate system: ˆ A = Q − 1 AQ q k → Q − 1 q k = e k span { q 1 , . . . , q j } → span { e 1 , . . . , e j } ready for next iteration
Subspace Iteration with changes of coordinate system Krylov subspaces David S. take S = span { e 1 , . . . , e j } Watkins p ( A ) S = span { p ( A ) e 1 , . . . , p ( A ) e j } = span { q 1 , . . . , q j } (orthonormal?) build unitary (or not) Q = [ q 1 · · · q j · · · ] change coordinate system: ˆ A = Q − 1 AQ q k → Q − 1 q k = e k span { q 1 , . . . , q j } → span { e 1 , . . . , e j } ready for next iteration
Subspace Iteration with changes of coordinate system Krylov subspaces David S. This version of subspace iteration . . . Watkins . . . holds the subspace fixed while the matrix changes. . . . moving toward a matrix under which span { e 1 , . . . , e j } is invariant. � A 11 � A 12 A → ( A 11 is j × j .) 0 A 22
Subspace Iteration with changes of coordinate system Krylov subspaces David S. This version of subspace iteration . . . Watkins . . . holds the subspace fixed while the matrix changes. . . . moving toward a matrix under which span { e 1 , . . . , e j } is invariant. � A 11 � A 12 A → ( A 11 is j × j .) 0 A 22
Subspace Iteration with changes of coordinate system Krylov subspaces David S. This version of subspace iteration . . . Watkins . . . holds the subspace fixed while the matrix changes. . . . moving toward a matrix under which span { e 1 , . . . , e j } is invariant. � A 11 � A 12 A → ( A 11 is j × j .) 0 A 22
Subspace Iteration with changes of coordinate system Krylov subspaces David S. This version of subspace iteration . . . Watkins . . . holds the subspace fixed while the matrix changes. . . . moving toward a matrix under which span { e 1 , . . . , e j } is invariant. � A 11 � A 12 A → ( A 11 is j × j .) 0 A 22
Subspace Iteration with changes of coordinate system Krylov subspaces David S. This version of subspace iteration . . . Watkins . . . holds the subspace fixed while the matrix changes. . . . moving toward a matrix under which span { e 1 , . . . , e j } is invariant. � A 11 � A 12 A → ( A 11 is j × j .) 0 A 22
Subspace iteration and Krylov subspaces Krylov subspaces David S. single vector q “determines” nested sequence Watkins � q , Aq , A 2 q , . . . , A j − 1 q � K j ( A , q ) = span , j = 1 , . . . , n step of power method: q → p ( A ) q . . . implies a nested sequence of subspace iterations because . . . p ( A ) K j ( A , q ) = K j ( A , p ( A ) q ), since p ( A ) A = Ap ( A )
Subspace iteration and Krylov subspaces Krylov subspaces David S. single vector q “determines” nested sequence Watkins � q , Aq , A 2 q , . . . , A j − 1 q � K j ( A , q ) = span , j = 1 , . . . , n step of power method: q → p ( A ) q . . . implies a nested sequence of subspace iterations because . . . p ( A ) K j ( A , q ) = K j ( A , p ( A ) q ), since p ( A ) A = Ap ( A )
Subspace iteration and Krylov subspaces Krylov subspaces David S. single vector q “determines” nested sequence Watkins � q , Aq , A 2 q , . . . , A j − 1 q � K j ( A , q ) = span , j = 1 , . . . , n step of power method: q → p ( A ) q . . . implies a nested sequence of subspace iterations because . . . p ( A ) K j ( A , q ) = K j ( A , p ( A ) q ), since p ( A ) A = Ap ( A )
Subspace iteration and Krylov subspaces Krylov subspaces David S. single vector q “determines” nested sequence Watkins � q , Aq , A 2 q , . . . , A j − 1 q � K j ( A , q ) = span , j = 1 , . . . , n step of power method: q → p ( A ) q . . . implies a nested sequence of subspace iterations because . . . p ( A ) K j ( A , q ) = K j ( A , p ( A ) q ), since p ( A ) A = Ap ( A )
Subspace iteration and Krylov subspaces Krylov subspaces David S. single vector q “determines” nested sequence Watkins � q , Aq , A 2 q , . . . , A j − 1 q � K j ( A , q ) = span , j = 1 , . . . , n step of power method: q → p ( A ) q . . . implies a nested sequence of subspace iterations because . . . p ( A ) K j ( A , q ) = K j ( A , p ( A ) q ), since p ( A ) A = Ap ( A )
Let’s build an algorithm Krylov subspaces David S. start with upper Hessenberg A Watkins pick shifts ρ 1 , ρ 2 ( m = 2 for illustration) x y z q 1 = α p ( A ) e 1 = α ( A − ρ 1 I )( A − ρ 2 I ) e 1 = 0 . . . 0 cheap, don’t compute p ( A ). � � Q 0 = q 1 · · · q n , built from q 1
Let’s build an algorithm Krylov subspaces David S. start with upper Hessenberg A Watkins pick shifts ρ 1 , ρ 2 ( m = 2 for illustration) x y z q 1 = α p ( A ) e 1 = α ( A − ρ 1 I )( A − ρ 2 I ) e 1 = 0 . . . 0 cheap, don’t compute p ( A ). � � Q 0 = q 1 · · · q n , built from q 1
Let’s build an algorithm Krylov subspaces David S. start with upper Hessenberg A Watkins pick shifts ρ 1 , ρ 2 ( m = 2 for illustration) x y z q 1 = α p ( A ) e 1 = α ( A − ρ 1 I )( A − ρ 2 I ) e 1 = 0 . . . 0 cheap, don’t compute p ( A ). � � Q 0 = q 1 · · · q n , built from q 1
Let’s build an algorithm Krylov subspaces David S. start with upper Hessenberg A Watkins pick shifts ρ 1 , ρ 2 ( m = 2 for illustration) x y z q 1 = α p ( A ) e 1 = α ( A − ρ 1 I )( A − ρ 2 I ) e 1 = 0 . . . 0 cheap, don’t compute p ( A ). � � Q 0 = q 1 · · · q n , built from q 1
Let’s build an algorithm Krylov subspaces David S. start with upper Hessenberg A Watkins pick shifts ρ 1 , ρ 2 ( m = 2 for illustration) x y z q 1 = α p ( A ) e 1 = α ( A − ρ 1 I )( A − ρ 2 I ) e 1 = 0 . . . 0 cheap, don’t compute p ( A ). � � Q 0 = q 1 · · · q n , built from q 1
Let’s build an algorithm Krylov subspaces David S. start with upper Hessenberg A Watkins pick shifts ρ 1 , ρ 2 ( m = 2 for illustration) x y z q 1 = α p ( A ) e 1 = α ( A − ρ 1 I )( A − ρ 2 I ) e 1 = 0 . . . 0 cheap, don’t compute p ( A ). � � Q 0 = q 1 · · · q n , built from q 1
Let’s build an algorithm Krylov subspaces David S. Watkins ˜ A = Q − 1 0 AQ 0 power method + change of coordinate system now return to Hessenberg form A = ˜ ˆ Q − 1 ˜ A ˜ Q = Q − 1 AQ iteration complete! Now repeat!
Let’s build an algorithm Krylov subspaces David S. Watkins ˜ A = Q − 1 0 AQ 0 power method + change of coordinate system now return to Hessenberg form A = ˜ ˆ Q − 1 ˜ A ˜ Q = Q − 1 AQ iteration complete! Now repeat!
Let’s build an algorithm Krylov subspaces David S. Watkins ˜ A = Q − 1 0 AQ 0 power method + change of coordinate system now return to Hessenberg form A = ˜ ˆ Q − 1 ˜ A ˜ Q = Q − 1 AQ iteration complete! Now repeat!
Let’s build an algorithm Krylov subspaces David S. Watkins ˜ A = Q − 1 0 AQ 0 power method + change of coordinate system now return to Hessenberg form A = ˜ ˆ Q − 1 ˜ A ˜ Q = Q − 1 AQ iteration complete! Now repeat!
Let’s build an algorithm Krylov subspaces David S. Watkins ˜ A = Q − 1 0 AQ 0 power method + change of coordinate system now return to Hessenberg form A = ˜ ˆ Q − 1 ˜ A ˜ Q = Q − 1 AQ iteration complete! Now repeat!
Let’s build an algorithm Krylov subspaces David S. Watkins ˜ A = Q − 1 0 AQ 0 power method + change of coordinate system now return to Hessenberg form A = ˜ ˆ Q − 1 ˜ A ˜ Q = Q − 1 AQ iteration complete! Now repeat!
What does the algorithm do? Krylov ˆ A = Q − 1 AQ subspaces David S. q 1 = α p ( A ) e 1 Watkins ˆ A Hessenberg, so span { q 1 , . . . , q j } = K j ( A , q 1 ) = p ( A ) K j ( A , e 1 ) = p ( A ) span { e 1 , . . . , e j } subspace iteration + change of coordinate system: span { q 1 , . . . , q j } → span { e 1 , . . . , e j } j = 1, 2, . . . , n − 1
What does the algorithm do? Krylov ˆ A = Q − 1 AQ subspaces David S. q 1 = α p ( A ) e 1 Watkins ˆ A Hessenberg, so span { q 1 , . . . , q j } = K j ( A , q 1 ) = p ( A ) K j ( A , e 1 ) = p ( A ) span { e 1 , . . . , e j } subspace iteration + change of coordinate system: span { q 1 , . . . , q j } → span { e 1 , . . . , e j } j = 1, 2, . . . , n − 1
What does the algorithm do? Krylov ˆ A = Q − 1 AQ subspaces David S. q 1 = α p ( A ) e 1 Watkins ˆ A Hessenberg, so span { q 1 , . . . , q j } = K j ( A , q 1 ) = p ( A ) K j ( A , e 1 ) = p ( A ) span { e 1 , . . . , e j } subspace iteration + change of coordinate system: span { q 1 , . . . , q j } → span { e 1 , . . . , e j } j = 1, 2, . . . , n − 1
What does the algorithm do? Krylov ˆ A = Q − 1 AQ subspaces David S. q 1 = α p ( A ) e 1 Watkins ˆ A Hessenberg, so span { q 1 , . . . , q j } = K j ( A , q 1 ) = p ( A ) K j ( A , e 1 ) = p ( A ) span { e 1 , . . . , e j } subspace iteration + change of coordinate system: span { q 1 , . . . , q j } → span { e 1 , . . . , e j } j = 1, 2, . . . , n − 1
What does the algorithm do? Krylov ˆ A = Q − 1 AQ subspaces David S. q 1 = α p ( A ) e 1 Watkins ˆ A Hessenberg, so span { q 1 , . . . , q j } = K j ( A , q 1 ) = p ( A ) K j ( A , e 1 ) = p ( A ) span { e 1 , . . . , e j } subspace iteration + change of coordinate system: span { q 1 , . . . , q j } → span { e 1 , . . . , e j } j = 1, 2, . . . , n − 1
What does the algorithm do? Krylov ˆ A = Q − 1 AQ subspaces David S. q 1 = α p ( A ) e 1 Watkins ˆ A Hessenberg, so span { q 1 , . . . , q j } = K j ( A , q 1 ) = p ( A ) K j ( A , e 1 ) = p ( A ) span { e 1 , . . . , e j } subspace iteration + change of coordinate system: span { q 1 , . . . , q j } → span { e 1 , . . . , e j } j = 1, 2, . . . , n − 1
What does the algorithm do? Krylov ˆ A = Q − 1 AQ subspaces David S. q 1 = α p ( A ) e 1 Watkins ˆ A Hessenberg, so span { q 1 , . . . , q j } = K j ( A , q 1 ) = p ( A ) K j ( A , e 1 ) = p ( A ) span { e 1 , . . . , e j } subspace iteration + change of coordinate system: span { q 1 , . . . , q j } → span { e 1 , . . . , e j } j = 1, 2, . . . , n − 1
What does the algorithm do? Krylov subspaces David S. Watkins Convergence rates: | p ( λ j +1 ) / p ( λ j ) | j = 1 , . . . , n − 1 All ratios matter.
What does the algorithm do? Krylov subspaces David S. Watkins Convergence rates: | p ( λ j +1 ) / p ( λ j ) | j = 1 , . . . , n − 1 All ratios matter.
Implementation Krylov subspaces David S. Watkins x y z q 1 = α p ( A ) e 1 = 0 . . . 0 (case m = 2)
Implementation Krylov subspaces David S. Watkins × × × × × × × × × × × × × × × × × × × × × × × × × ×
Implementation Krylov subspaces David S. Watkins × × × × × × × × × × × × × × × × × × × × × × × × × × ×
Implementation Krylov subspaces David S. Watkins × × × × × × × × × × × × × × × × × × × × × × × × × × × × ×
Implementation Krylov subspaces David S. Watkins × × × × × × × × × × × × × × × × × × × × × × × × × × × × ×
Implementation Krylov subspaces David S. Watkins × × × × × × × × × × × × × × × × × × × × × × × × × × × × ×
Implementation Krylov subspaces David S. Watkins × × × × × × × × × × × × × × × × × × × × × × × × × × ×
Implementation Krylov subspaces David S. Watkins × × × × × × × × × × × × × × × × × × × × × × × × × ×
Implementation Krylov subspaces David S. Watkins O ( n 2 ) work per iteration (cheap!) This is nothing new. John Francis’s algorithm aka implicitly-shifted QR algorithm
Implementation Krylov subspaces David S. Watkins O ( n 2 ) work per iteration (cheap!) This is nothing new. John Francis’s algorithm aka implicitly-shifted QR algorithm
Implementation Krylov subspaces David S. Watkins O ( n 2 ) work per iteration (cheap!) This is nothing new. John Francis’s algorithm aka implicitly-shifted QR algorithm
Implementation Krylov subspaces David S. Watkins O ( n 2 ) work per iteration (cheap!) This is nothing new. John Francis’s algorithm aka implicitly-shifted QR algorithm
Recommend
More recommend