The QR algorithm is a method for calculating all eigenvalues We - PowerPoint PPT Presentation

Lesson 22 QR A LGORITHM

• The QR algorithm is a method for calculating all eigenvalues • We will see that the pure QR algorithm is equivalent to power iteration applied to multiple vectors at once • It therefore suffers the same problems as power iteration • To make the algorithm practical, we use shifts, like in Rayleigh iteration • We also reduce matrices to tridiagonal form • We also introduce the Schur decomposition and calculate SVD

S IMULTANEOUS ITERATION

• Suppose we want to find the two largest eigenvalues λ 1 and λ 2 • The idea is to run power iteration on two column vectors simultaneously � � v 1 | v 2 V 0 = V k = AV k − 1 • Now each normalized column converges to the same vector q 1 • But we claim that the span of the columns ���� converges in a sense to the span of the two largest eigenvectors ���� • In other words, where

• Suppose we want to find the two largest eigenvalues λ 1 and λ 2 • The idea is to run power iteration on two column vectors simultaneously � � v 1 | v 2 V 0 = V k = AV k − 1 • Now each normalized column converges to the same vector q 1 • But we claim that the span of the columns ���� V k converges in a sense to the span of the two largest eigenvectors ���� � � q 1 | q 2 • In other words, Q k → where � � ± q 1 | ± q 2 Q k R k = V k

• We can make this precise: we have � � c 11 c 12 . . � � n j =1 c j 1 q j | � n � . . V 0 = j =1 c j 2 q j = Q � � . . � � c n 1 c n 2 = QC • Then we have, since • Assuming that is nonsingular, we have, without changing the column space where are the entries of

• We can make this precise: we have � � c 11 c 12 . . � � n j =1 c j 1 q j | � n � . . V 0 = j =1 c j 2 q j = Q � � . . � � c n 1 c n 2 = QC • Then we have, since AQ = Q Λ V k = A k QC = Q Λ k C � � • Assuming that is nonsingular, we have, without changing the column space where are the entries of

• We can make this precise: we have � � c 11 c 12 . . � � n j =1 c j 1 q j | � n � . . V 0 = j =1 c j 2 q j = Q � � . . � � c n 1 c n 2 = QC • Then we have, since AQ = Q Λ V k = A k QC = Q Λ k C � � c 11 c 12 • Assuming that C 1:2 , 1:2 := is nonsingular, we have, without changing c 21 c 22 the column space � � � � 1 0 1 0 0 1 0 1 � � � � λ − k � � � � � � ( λ 2 / λ 1 ) k ˜ ( λ 2 / λ 1 ) k ˜ 0 0 c 31 c 31 V k C − 1 1 = Q → Q � � � � λ − k 1:2 , 1:2 � . . � � . . � . . . . 2 � � � � . . . . � � � � ( λ n / λ 1 ) k ˜ ( λ 2 / λ 1 ) k ˜ 0 0 c n 1 c n 1 where ˜ c kj are the entries of C − 1 1:2 , 1:2

• A more numerically practical approach is to orthogonalize at each step: V 0 = QC Q k ¯ ¯ R k = V k V k = A ¯ Q k − 1 • This is the same as or in other words, we are calculating the QR decomposition of : • Note that we can easily apply this to 3, 4, or even columns

• A more numerically practical approach is to orthogonalize at each step: V 0 = QC Q k ¯ ¯ R k = V k V k = A ¯ Q k − 1 • This is the same as V k = A ¯ Q k − 1 = A ¯ Q k − 1 ¯ R k − 1 ¯ k − 1 = AV k − 1 ¯ R − 1 R − 1 k − 1 = · · · = A k V 0 ¯ · · · ¯ R − 1 R − 1 0 k − 1 or in other words, we are calculating the QR decomposition of : • Note that we can easily apply this to 3, 4, or even columns

• A more numerically practical approach is to orthogonalize at each step: V 0 = QC Q k ¯ ¯ R k = V k V k = A ¯ Q k − 1 • This is the same as V k = A ¯ Q k − 1 = A ¯ Q k − 1 ¯ R k − 1 ¯ k − 1 = AV k − 1 ¯ R − 1 R − 1 k − 1 = · · · = A k V 0 ¯ · · · ¯ R − 1 R − 1 0 k − 1 or in other words, we are calculating the QR decomposition of A k V 0 : A k V 0 = V k ¯ R k − 1 · · · ¯ R 0 = ¯ Q k ¯ R k ¯ R k − 1 · · · ¯ R 0 = ¯ Q k ˜ R k • Note that we can easily apply this to 3, 4, or even n columns

QR A LGORITHM ( NOT DECOMPOSITION )

���� • The pure QR Algorithm consists of calculating a QR decomposition and recombining in the other order: for A 0 = A , define Q k R k = A k � 1 A k = R k Q k • Note that has the same eigenvalue as • The remarkable fact is that, provided the eigenvalues are strictly ordered ( ) then this converges to a diagonal matrix:

���� • The pure QR Algorithm consists of calculating a QR decomposition and recombining in the other order: for A 0 = A , define Q k R k = A k � 1 A k = R k Q k • Note that A k = R k Q k = Q � k A k � 1 Q k has the same eigenvalue as A • The remarkable fact is that, provided the eigenvalues are strictly ordered ( ) then this converges to a diagonal matrix:

• The pure QR Algorithm consists of calculating a QR decomposition and recombining in the other order: for A 0 = A , define Q k R k = A k � 1 A k = R k Q k • Note that A k = R k Q k = Q � k A k � 1 Q k has the same eigenvalue as A • The remarkable fact is that, provided the eigenvalues are strictly ordered ( | λ 1 | > | λ 2 | > · · · > | λ n | ) then this converges to a diagonal matrix: A k → ���� { λ 1 , . . . , λ n }

• Consider a randoom symmetric Gaussian matrix � A k � ���� A k �   G 11 G 1 n · · · 1 . . ... . . G n =   . . √ n   0.01 G 1 n G nn · · · 10 - 5 where G ij are random Gaussian entries 10 - 8 � mean zero and variance twice on the diagonal as on the off diagonal 10 - 11 k 5000 10000 15000 20000

QR A LGORITHM = S IMULTANEOUS ITERATION

• We use the QR decomposition with R ii > 0 , which is unique (via Gram–Schmidt) • Simultaneous iteration on n vectors with initial guess of I is ¯ Q 0 = I Z k = A ¯ Q k � 1 Q k ¯ ¯ R k = Z k • QR algorithm is • We claim that and

• We use the QR decomposition with R ii > 0 , which is unique (via Gram–Schmidt) • Simultaneous iteration on n vectors with initial guess of I is ¯ Q 0 = I Z k = A ¯ Q k � 1 Q k ¯ ¯ R k = Z k • QR algorithm is A 0 = A Q k R k = A k � 1 A k = R k Q k • We claim that and

• We use the QR decomposition with R ii > 0 , which is unique (via Gram–Schmidt) • Simultaneous iteration on n vectors with initial guess of I is ¯ Q 0 = I Z k = A ¯ Q k � 1 Q k ¯ ¯ R k = Z k • QR algorithm is A 0 = A Q k R k = A k � 1 A k = R k Q k • We claim that ¯ ¯ A k = ¯ k A ¯ and Q � Q k = Q 1 · · · Q k , R k = R k Q k

• QR algorithm converges provided that Q � = LU exists � The proof is straightforward when the eigenvalues are strictly ordered | λ 1 | > | λ 2 | > · · · > | λ n | • Importantly, the ( n, n ) entry converges to the smallest eigenvalue � We only care about convergence of smallest eigenvalue, as we can use shifts to make a small eigenvalue smaller and accelerate convergence

R EDUCTION TO TRIDIAGONAL FORM

• Suppose A is symmetric tridiagonal   α 1 β 1 β 1 α 2 β 2     ... ... ... A =       β n � 2 α n � 1 β n � 1   β n � 1 α n • Consider the sequence of Given's rotations that introduce zeros one at a time, which we saw introduces one superdiagonal: • acts only on the and th rows when acting on the left, so acts only on the and th columns when acting on the right • So the next stage in the QR algorithm has only one subdiagonal:

• Suppose A is symmetric tridiagonal   α 1 β 1 β 1 α 2 β 2     ... ... ... A =       β n � 2 α n � 1 β n � 1   β n � 1 α n • Consider the sequence of Given's rotations that introduce zeros one at a time, which we saw introduces one superdiagonal:   × × × × × ×     ... ... ...   R = Q � A = Q � n � 1 · · · Q � 1 A =     × × ×     × ×   × • acts only on the and th rows when acting on the left, so acts only on the and th columns when acting on the right • So the next stage in the QR algorithm has only one subdiagonal:

• Suppose A is symmetric tridiagonal   α 1 β 1 β 1 α 2 β 2     ... ... ... A =       β n � 2 α n � 1 β n � 1   β n � 1 α n • Consider the sequence of Given's rotations that introduce zeros one at a time, which we saw introduces one superdiagonal:   × × × × × ×     ... ... ...   R = Q � A = Q � n � 1 · · · Q � 1 A =     × × ×     × ×   × • Q � k acts only on the k and ( k +1) th rows when acting on the left, so Q k acts only on the k and ( k + 1) th columns when acting on the right • So the next stage in the QR algorithm has only one subdiagonal:   × × × × × · · · × × × × × · · ·     × × × × · · ·   RQ = RQ 1 · · · Q n =  . .  ... ... . .   . .     × × ×   × ×