Systems of Linear Equations The purpose of computing is insight, not - PowerPoint PPT Presentation

Systems of Linear Equations The purpose of computing is insight, not numbers. Richard Wesley Hamming 1

Problem Description: 1/2 � Solving systems of linear equations is one of the most important tasks in numerical methods. � The i -the equation (1 ≤ i ≤ n ) is a i 1 x 1 + a i 2 x 2 + a i 3 x 3 + …… + a in x n = b i , where a i 1 , a i 2 , a i 3 , ……, a in and b i are known values, and the x i ’s are unknowns to be solved from the n linear equations. a 11 x 1 + a 12 x 2 + a 13 x 3 + …… + a 1 n x n = b 1 a 21 x 1 + a 22 x 2 + a 23 x 3 + …… + a 2 n x n = b 2 ……………………………………… a n 1 x 1 + a n 2 x 2 + a n 3 x 3 + …… + a nn x n = b n 2

Problem Description: 2/2 � A system of linear equations is usually represented by matrices, A=[ a ij ] n × n the coefficient matrix, [ b i ] n × 1 the constant matrix, and [ x i ] n × 1 the unknown matrix. ⎡ � ⎤ ⎡ ⎤ ⎡ ⎤ a a a x b 11 12 1 1 1 n ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ � a a a x b ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ 21 22 2 2 2 n ⋅ = ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ � � � � � � ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ � ⎣ ⎦ ⎣ ⎦ ⎣ ⎦ a a a x b 1 2 n n nn n n 3

Methods to Be Discussed � Methods for solving systems of linear equations are of two types, direct and iterative . � Direct methods go through a definite procedure to find the solution. Elimination and decomposition are the two major approaches. � Iterative methods, which is similar to solving non-linear equations, find the solution iteratively. 4

Gaussian Elimination: 1/6 � Suppose all conditions are ideal ( i.e ., no problem will occur during the computation process). � Gaussian elimination is very effective and easy to implement. � Idea-1: Idea-1: The idea is very simple. Use the i -th equation to eliminate unknown x i from equations i +1 to n . This is the elimination stage. � Idea-2: Idea-2: After the elimination stage is done, a backward substitution is performed to find the solutions. 5

Gaussian Elimination: 2/6 � Consider the following simple elimination example. Use equation 1 to eliminate variable x in equations 2 and 3. x – y + z = 3 × (-2)+ × (-3)+ 2x + y – z = 0 3x + 2y + 2z = 15 x – y + z = 3 3y – 3z = -6 5y – z = 6 x is eliminated from equations 2 and 3 6

Gaussian Elimination: 3/6 � Use equation 2 to eliminate y in equation 3. � After elimination, the system is upper triangular ! x – y + z = 3 3y – 3z = -6 × (-5/3)+ 5y – z = 6 x – y + z = 3 3y – 3z = -6 4z = 16 y is eliminated from equation 3 Equation 2 only has z ! 7

Gaussian Elimination: 4/6 � Now we can solve for z from equation 3. � Plug z into equation 2 to solve for y . � Then, plug y and z into equation 1 to solve for x . � This is backward substitution ! x – y + z = 3 x = 3+y-z 3y – 3z = -6 y = (-6+3z)/3 4z = 16 z = 16/4 8

Gaussian Elimination: 5/6 � Each entry of row i is multiplied by – a k , i / a i , i and added to row k , for all k with i +1 ≤ k ≤ n . � In this way, all entries on column i below a i , i are eliminated ( i.e ., zero). Do the same for b i ’s. � � a a a a 11 1 , 1 , 1 1 i i + n � � � � 0 a a a × (- a i +1, i / a i , i )+ × (- a n , i / a i , i )+ , , 1 , i i i i + i n � 0 a a a 1 , 1 , 1 1 , i + i i + i + i + n � � � � � � 0 a a a , , 1 , n i n i + n n 9

Gaussian Elimination: 6/6 � After elimination, the equations become upper triangular , i.e ., all lower diagonal entries being 0’s. � � a a a a 11 1 , 1 , 1 1 i i + n � � � � 0 a a a , , 1 , i i i i + i n � 0 0 a a All Zeros 1 , 1 1 , i + i + i + n � � � � � � 0 0 0 a n n , � Equation i only has variables x i , x i +1 , …, x n : 1 � a x + a x + + a x = b , , 1 , i i i i i i i n n n + + 10

Backw ard Substitution � Equation n is a n , n x n = b n , and x n = b n / a n,n . � Equation n -1 is a n -1, n -1 x n -1 + a n -1, n x n = b n -1 , and x n -1 = ( b n -1 - a n -1, n x n )/ a n -1, n -1 � Equation i is and x i is 1 � a x + a x + + a x = b , , 1 , i i i i i + i + i n n n computed as: ⎛ ⎞ n 1 ∑ ⎜ ⎟ x = b − a x ⎝ ⎠ , i i i k k a 1 , k i = + i i DO i = n, 1, -1 ! Going backward S = b(i) ! Initial S to b(i) DO k = i+1, n ! Sum all terms to the right S = S – a(i,k)*x(k) ! of a(i,i) END DO x(i) = S/a(i,i) ! Compute x(i) END DO 11

How to Eliminate? � Gaussian Elimination � � a a a a , , 1 , , i i i i + i j i n � � � uses row i ( i.e ., a i , i ) to � � � eliminates a k , i for k > i . � � a a a a � Thus, all entries below , , 1 , , k i k i + k j k n a i , i become zero! zero here DO i = 1, n-1 ! Using row i, i.e., a(i,i) DO k = i+1, n ! We want to eliminate a(k,i) S = -a(k,i)/a(i,i) DO j = i+1, n ! for each entry on row k a(k,j) = a(k,j) + S*a(i,j) ! update its value END DO a(k,i) = 0 ! kill a(k,i) b(k) = b(k) + S*b(i) END DO ! Don’t forget to update b(k) 12 END DO

Efficiency Issues: 1/4 � How to determine the efficiency of Gaussian elimination? � We count the number of multiplications and divisions, since multiplications and divisions are slower than additions and subtractions. � As long as we know the dominating factor, we know the key of efficiency. � Since divisions are not used frequently, counting multiplications would be good enough. 13

Efficiency Issues: 2/4 � Fixing i , the inner-most j loop executes n - i times, and uses n - i multiplications. One more is needed to update b k . Therefore, the total is n - i +1. � Since k goes from i +1 to n and the inner-most j loop executes n - i times, there are ( n - i )( n - i +1) multiplications. 1 n − ∑ � Since i goes from 1 to n -1, the total is ( )( 1 ) n − i n − + i 1 i = DO i = 1, n-1 DO k = i+1, n S = -a(k,i)/a(i,i) DO j = i+1, n a(k,j) = a(k,j) + S*a(i,j) END DO a(k,j) = 0 b(k) = b(k) + S*b(i) END DO 14 END DO

Efficiency Issues: 3/4 � The following can be proved with induction: 1 � n 1 2 ( 1 ) n n + + = + 2 1 ( ) � n 2 2 2 2 3 2 1 2 3 6 2 3 + + + + = n + n + n � Therefore, we have (verify yourself): 1 1 1 n − n − n − 1 ∑ ∑ ∑ ( ) 2 3 ( )( 1 ) ( ) ( ) n − i n − + i = n − i + n − i = n − n 3 1 1 1 i = i = i − � This is an O ( n 3 ) algorithm ( i.e ., number of multiplications proportional to n 3 ). 15

Efficiency Issues: 4/4 � In the backward substitution step, since we need to compute the sum a i , i +1 x i +1 + a i , i +2 x i +1 +…+ a i , n x n , n - i multiplications are needed. � Since i runs from n -1 down to 1, the total number 1 n − 1 ∑ of multiplications is ( ) ( 1 ) n − i = n n − 2 i 1 = � In summary, the total number of multiplications used is dominated by the elimination step, which is proportional to n 3 /3, or O ( n 3 ) ! � Exercise: How many divisions are needed in the elimination and the backward substitution stages? 16

A Serious Problem � There is a big big problem. When eliminating a k , i using a i , i , – a k , i / a i , i is computed. � What if a i , i ≈ 0? Division-by-zero or overflow in – a k , i / a i , i or truncation in (– a k , i / a i , i ) × a i , j + a k , j may occur because of (– a k , i / a i , i ) × a i,j >> a k,j ! Why? Why? � Therefore, the use of Gaussian elimination is risky and some modifications would be needed. � However, if the following holds ( i.e ., diagonal dominant ), Gaussian elimination works fine. n ∑ | | | | a > a (for every i ) , , i i i j 17 1 , j = j i ≠

Partial Pivoting: 1/6 � To overcome the possible x /0 issue, one may use an important technique: pivoting . � There are two types of pivoting, partial (or row ) and complete . For most cases, partial pivoting usually works efficiently and accurately. � Important Observation Important Observation: Swapping two equations does not change the solutions! � If a i , i ≈ 0, one may swap some equation k with equation i so that the new a i , i would not be zero. � But, which k should be swapped? 18

Partial Pivoting: 2/6 � One of the best candidates is the a k , i such that | a k , i | is the maximum of a i , i , a i +1, i , …, a n , i . � Why? Why? If equation k with maximum | a k , i | is swapped to equation i , then, all | a j , i / a i , i | ≤ 1 for i ≤ j ≤ n . � � a a a a 11 1 , 1 , 1 1 i i + n � � � � 0 a a a , , 1 , i i i i + i n � 0 a a a 1 , 1 , 1 1 , i + i i + i + i + n find the maximum here � � � � followed by a swap � � 0 a a a , , 1 , n i n i + n n 19