Linear Systems CS3220 Summer 2008 Jonathan Kaldor Systems of - PowerPoint PPT Presentation

Linear Systems CS3220 Summer 2008 Jonathan Kaldor

Systems of Linear Equations • Want to find x, y, z such that 2x + 2z = 6 y - 3z = 4 2x + 4y = 2 • We have 3 linear equations, and 3 unknowns

Systems of Linear Equations • More formally, a system of n linear equations in n variables is: f 1 (x 1 , x 2 , ..., x n ) = b 1 f 2 (x 1 , x 2 , ..., x n ) = b 2 ... f n (x 1 , x 2 , ..., x n ) = b n • where each of the f i is linear in each of the unknowns x i

What makes it linear? • Formally, a function is linear iff f( x + x ’) = f( x ) + f( x ’) f(c x ) = c f( x ) • Functions that satisfy above are of the form f(x 1 , x 2 , ..., x n ) = a 1 x 1 + a 2 x 2 +...+ a n x n • Note: can also express as f( x ) = a T x where x and a are vectors

What makes it linear?

What makes it linear? • Important! A linear function must be linear in the unknowns • Take sin(a 1 ) x 1 + cos(a 2 ) x 2 • Is it linear if a 1 , a 2 are unknown? • Is it linear if x 1 , x 2 are unknown?

What makes it linear? • Important! A linear function must be linear in the unknowns • Take sin(a 1 ) x 1 + cos(a 2 ) x 2 • Is it linear if a 1 , a 2 are unknown? • Is it linear if x 1 , x 2 are unknown? • Let b 1 = sin(a 1 ), b 2 = cos(a 2 ), then above is b 1 x 1 + b 2 x 2

Our Example • Recall we want to find x, y, z such that: 2x + 2z = 6 y - 3z = 4 2x + 4y = 2 • Rewrite as 2x + 0y + 2z = 6 0x + 1y + -3z = 4 2x + 4y + 0z = 2

Our Example • Recall we want to find x, y, z such that: 2x + 2z = 6 y - 3z = 4 2x + 4y = 2 • Rewrite as 2x + 0y + 2z = 6 2 0 2 x 6 0x + 1y + -3z = 4 0 1 -3 y = 4 2x + 4y + 0z = 2 2 4 0 z 2

More generally

More generally • With n equations in n unknowns: a 11 a 12 .... a 1n x 1 b 1 a 21 a 22 .... a 2n x 2 b 2 . . . . . . = . . . . . . . . . . . . a n1 a n2 .... a nn x n b n

More generally • With n equations in n unknowns: a 11 a 12 .... a 1n x 1 b 1 a 21 a 22 .... a 2n x 2 b 2 . . . . . . = . . . . . . . . . . . . a n1 a n2 .... a nn x n b n • Ax = b

Solutions • The system Ax = b can have one solution, no solution, or infintely many solutions • If A is nonsingular, guaranteed to have one solution • From now on, assume A is nonsingular

Examples / Applications • Linear systems extremely common • Partly due to convenience • Used in one way or another in most of the other topics we will discuss

Circuit Diagrams • Given R j and V k , find i 1 , i 2 , i 3 R 1 R 2 i 1 i 2 R 3 V 1 i 3 V 2 R 5 R 4

Circuit Diagrams • Equations follow from Kirchoff’s Laws: • current is conserved • voltage losses = voltage gains around loops in circuit

Linear Algebra • Given Ax = b , solution is x = A -1 b • However, in practice we rarely, if ever, form the inverse • Would like to convert this into an easier problem to solve without changing the answer

Solving Linear Systems • Observation: Diagonal systems are easy to solve

Solving Linear Systems • Observation: Diagonal systems are easy to solve a 11 0 .... 0 x 1 b 1 0 a 22 .... 0 x 2 b 2 . . . . . . = . . . . . . . . . . . . 0 0 .... a nn x n b n

Linear Algebra Redux • We can scale any equation, or add any multiple of an equation to any other • Matrix formulation: scale row, add multiple of one row to another row • Need to also perform operation on right hand side

Gauss-Jordan Elimination • Take first equation, scale and subtract from all other equations in order to eliminate first variable in equations 2...n • Repeat with second equation and second variable... • does this affect first variable? • After n steps, have diagonal system

Our Example Scale by -1 2 0 2 x 1 6 and add to Eqn 3 0 1 -3 x 2 = 4 2 4 0 x 3 2

Our Example Scale by -1 2 0 2 x 1 6 and add to Eqn 3 0 1 -3 x 2 = 4 0 4 -2 x 3 -4

Our Example 2 0 2 x 1 6 Scale by -4 0 1 -3 x 2 = 4 and add to Eqn 3 0 4 -2 x 3 -4

Our Example 2 0 2 x 1 6 Scale by -4 0 1 -3 x 2 = 4 and add to Eqn 3 0 0 10 x 3 -20

Our Example 2 0 2 x 1 6 0 1 -3 x 2 = 4 Scale by -2/10 0 0 10 x 3 -20 and add to Eqn 1

Our Example 2 0 0 x 1 10 0 1 -3 x 2 = 4 Scale by -2/10 0 0 10 x 3 -20 and add to Eqn 1

Our Example 2 0 0 x 1 10 0 1 -3 x 2 = 4 Scale by 3/10 0 0 10 x 3 -20 and add to Eqn 2

Our Example 2 0 0 x 1 10 0 1 0 x 2 = -2 Scale by 3/10 0 0 10 x 3 -20 and add to Eqn 2

Our Example 2 0 0 x 1 10 Scale each row so 0 1 0 x 2 = -2 diagonal entry is 1 0 0 10 x 3 -20

Our Example 1 0 0 x 1 5 Scale each row so 0 1 0 x 2 = -2 diagonal entry is 1 0 0 1 x 3 -2

Our Example 1 0 0 x 1 5 Scale each row so 0 1 0 x 2 = -2 diagonal entry is 1 0 0 1 x 3 -2 Answer is x 1 =5, x 2 =-2, x 3 =-2

Matrix Justification • Want to make solving Ax = b easier • Suppose solution to MAx = Mb for some choice of M is x ’ • Then if M -1 exists, x ’ is obviously a solution to Ax = b as well. • Want to find non-singular M such that MAx = Mb is easier to solve

Matrix Justification • Matrix M j adds multiple of jth row to every other row • What multiple?

Matrix Justification • Matrix M j adds multiple 1 0 0 ... -a 1j /a jj ... 0 of jth row to every other row 0 1 0 ... -a 2j /a jj ... 0 . . . . . . . • What multiple? . . . . . . . 0 0 0 ... -1/a jj ... 0 . . . . . . . . . . . . . . 0 0 0 ... -a nj /a jj ... 1

Matrix Justification • Note: Never really need to form the matrices M • We know what happens when we multiply by M • Faster to just perform the scales and additions • How much faster?

Matrix Justification • At the end, M n-1 ... M 2 M 1 Ax = M n-1 ... M 2 M 1 b • Left hand side is the identity matrix • But that means M n-1 ... M 2 M 1 is the inverse of A • Again, dont need to form M i • Just apply row operations to identity

Multiple Right Hand Sides • Suppose we have several sets of equations, each with the same coefficients but different RHS • Matrix notation: Ax = b 1 , Ax = b 2 , Ax = b 3 , ... Ax = b m • Can just stack b i into n x m B matrix • Apply M i to matrix B , get answer X

Triangular Matrices • Don’t have to get it all the way diagonal to solve easily • Observation: once we know a variable’s value, we can substitute it in all other equations • If that substitution allows us to solve for another variable...

Triangular Matrices

Triangular Matrices • Upper Triangular Matrix a 11 a 12 .... a 1n x 1 b 1 0 a 22 .... a 2n x 2 b 2 . . . . . . = . . . . . . . . a n-1,n-1 a n-1,n x n-1 b n-1 0 0 .... a nn x n b n

Triangular Matrices • Upper Triangular Matrix a 11 a 12 .... a 1n x 1 b 1 0 a 22 .... a 2n x 2 b 2 . . . . . . = . . . . . . . . a n-1,n-1 a n-1,n x n-1 b n-1 0 0 .... a nn x n b n Solve for x n

Triangular Matrices • Upper Triangular Matrix a 11 a 12 .... a 1n x 1 b 1 0 a 22 .... a 2n x 2 b 2 Substitute x n , . . . . . . = solve for x n-1 . . . . . . . . a n-1,n-1 a n-1,n x n-1 b n-1 0 0 .... a nn x n b n Solve for x n

Triangular Matrices • Upper Triangular Matrix Repeat for each variable a 11 a 12 .... a 1n x 1 b 1 0 a 22 .... a 2n x 2 b 2 Substitute x n , . . . . . . = solve for x n-1 . . . . . . . . a n-1,n-1 a n-1,n x n-1 b n-1 0 0 .... a nn x n b n Solve for x n

Triangular Matrices • Upper Triangular Matrix Repeat for each variable a 11 a 12 .... a 1n x 1 b 1 0 a 22 .... a 2n x 2 b 2 Substitute x n , . . . . . . = solve for x n-1 . . . . . . . . a n-1,n-1 a n-1,n x n-1 b n-1 0 0 .... a nn x n b n Solve for x n • “Backward Substitution”

Triangular Matrices • Forward Substitution: Corresponding method for lower triangular matrices • MATLAB code for backward substitution

Gaussian Elimination • Instead of reducing to diagonal matrix, reduce to upper triangular system • What do M i matrices look like now? • What do M i-1 look like? • What does ( M n-1 ... M 2 M 1 ) -1 = M look like?

LU Factorization • Convert Ax = b into MA x= Mb , where MA is upper triangular • M is lower triangular, as is M -1 • Let MA = U , M -1 = L • Then LU = M -1 MA = A • So Ax = b can be converted to LUx = b • L lower triangular, U upper triangular

Solving with LU factorization • Have LUx = b • Solve Ly = b (forward substitution) • Then solve Ux = y (backward substitution) • Note: Don’t need to have b in order to find LU factorization • Can factor and then solve for many RHS