Overview Last time we introduced the Gram Schmidt process as an algorithm for turning a basis for a subspace into an orthogonal basis for the same subspace. Having an orthogonal basis (or even better, an orthonormal basis!) is helpful for many problems associated to orthogonal projection. Today we’ll discuss the “Least Squares Problem", which asks for the best approximation of a solution to a system of linear equations in the case when an exact solution doesn’t exist. From Lay, §6.5 Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 1 / 21
1. Introduction Problem : What do we do when the matrix equation A x = b has no solution x ? Such inconsistent systems A x = b often arise in applications, sometimes with large coefficient matrices. Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 2 / 21
1. Introduction Problem : What do we do when the matrix equation A x = b has no solution x ? Such inconsistent systems A x = b often arise in applications, sometimes with large coefficient matrices. Answer : Find ˆ x such that A ˆ x is as close as possible to b . In this situation A ˆ x is an approximation to b . The general least squares problem is to find an ˆ x that makes � b − A ˆ x � as small as possible. Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 2 / 21
Definition For an m × n matrix A , a least squares solution to A x = b is a vector ˆ x such that for all x in R n . � b − A ˆ x � ≤ � b − A x � The name “least squares” comes from � · � 2 being the sum of the squares of the coordinates. Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 3 / 21
Definition For an m × n matrix A , a least squares solution to A x = b is a vector ˆ x such that for all x in R n . � b − A ˆ x � ≤ � b − A x � The name “least squares” comes from � · � 2 being the sum of the squares of the coordinates. It is now natural to ask ourselves two questions: (1) Do least square solutions always exist? The answer to this question is YES. We will see that we can use the Orthogonal Decomposition Theorem and the Best Approximation Theorem to show that least square solutions always exist. (2) How can we find least square solutions? The Orthogonal Decomposition Theorem —and in particular, the uniqueness of the orthogonal decomposition— gives a method to find all least squares solutions. Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 3 / 21
Solution of the general least squares problem � � Consider an m × n matrix A = a 1 a 2 a n . . . . x 1 x 2 is a vector in R n , then the definition of matrix-vector If x = . . . x n multiplication implies that A x = x 1 a 1 + x 2 a 2 + · · · + x n a n . So, the vector A x is the linear combination of the columns of A with weights given by the entries of x . For any vector x in R n that we select, the vector A x is in Col A . We can solve A x = b if and only if b is in Col A . Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 4 / 21
If the system A x = b is inconsistent it means that b is NOT in Col A . So we seek ˆ x that makes A ˆ x the closest point in Col A to b . The Best Approximation Theorem tells us that the closest point in Col A to b is ˆ b = proj Col A b . x = ˆ So we seek ˆ x such that A ˆ b . In other words, the least squares solutions of A x = b are exactly the solutions of the system x = ˆ A ^ b . x = ˆ By construction, the system A ˆ b is always consistent. Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 5 / 21
We seek ˆ x such that A ˆ x is the closest point to b in Col A . Equivalently, we need to find ˆ x with the property that A ˆ x is the orthogonal projection of b onto Col( A ). Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 6 / 21
Since ˆ x = ˆ b is the closest point to b in Col A , we need ˆ x such that A ˆ b . Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 7 / 21
The normal equations By the Orthogonal Decomposition Theorem, the projection ˆ b is the unique vector in Col A with the property that b − ˆ b is orthogonal to Col A . x in R n the vector A ˆ Since for every ˆ x is automatically in Col A , x = ˆ requiring that A ˆ b is the same as requiring that b − A ˆ x is orthogonal to Col A . This is equivalent to requiring that b − A ˆ x is orthogonal to each column of A . This means a T x ) = 0 , a T x ) = 0 , · · · , a T 1 ( b − A ˆ 2 ( b − A ˆ n ( b − A ˆ x ) = 0 . This gives a T 0 1 a T 0 2 ( b − A ˆ x ) = . . . . . . a T 0 n A T ( b − A ˆ x ) = 0 A T b − A T A ˆ x = 0 Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 8 / 21
A T A ˆ x = A T b These are the normal equations for ˆ x . Theorem The set of least-squares solutions of A x = b coincides with the nonempty set of solutions of the normal equations A T A ˆ x = A T b . Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 9 / 21
x is automatically in Col A and ˆ Since A ˆ b is the unique vector in Col A such that b − ˆ x = ˆ b is orthogonal to Col A , requiring that A ˆ b is the same as requiring that b − A ˆ x is orthogonal to Col A . Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 10 / 21
Examples Example 1 Find a least squares solution to the inconsistent system A x = b , where 1 3 5 A = 1 − 1 and b = 1 . 1 1 0 Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 11 / 21
Examples Example 1 Find a least squares solution to the inconsistent system A x = b , where 1 3 5 A = 1 − 1 and b = 1 . 1 1 0 To solve the normal equations A T A ˆ x = A T b , we first compute the relevant matrices: � � � � 1 3 1 1 1 3 3 A T A = 1 − 1 = 3 − 1 1 3 11 1 1 Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 11 / 21
� � � � 5 1 1 1 6 A T b = 1 = 3 − 1 1 14 . 0 � � � � 3 3 6 So we need to solve ˆ x = . The augmented matrix is 3 11 14 � � � � � � � � � 3 3 6 1 1 2 1 1 2 1 1 2 1 0 → → → → 3 11 14 3 11 14 0 8 8 0 1 1 0 1 � � 1 This gives ˆ x = . 1 � � 1 3 4 1 Note that A ˆ x = 1 − 1 = 0 and this is the closest point in Col A 1 1 1 2 5 to b = 1 . 0 Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 12 / 21
� � 3 3 We could note in this example that A T A = is invertible with 3 11 � � inverse 1 11 − 3 . In this case the normal equations give 24 − 3 3 A T A ˆ x = A T b ⇐ x = ( A T A ) − 1 A T b . ⇒ ˆ So we can calculate ( A T A ) − 1 A T b ˆ x = � � � � 1 11 − 3 6 = − 3 3 14 24 � � 1 = 1 . Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 13 / 21
Example 2 Find a least squares solution to the inconsistent system A x = b , where 3 − 1 4 A = 1 − 2 and b = 3 . 2 3 2 Notice that � � � � 3 − 1 3 1 2 14 1 A T A = 1 − 2 = is invertible. Thus the − 1 − 2 3 1 14 2 3 normal equations become A T A ˆ A T b x = ( A T A ) − 1 A T b ˆ x = Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 14 / 21
Furthermore, � � � � 4 3 1 2 19 A T b = 3 = − 1 − 2 3 − 4 2 So in this case ( A T A ) − 1 A T b � x = � � − 1 � � 14 1 19 = 1 14 − 4 � � � � 1 14 − 1 19 = 195 − 1 14 − 4 � � 1 18 = 13 − 5 . Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 15 / 21
With these values, we have 59 5 . 54 x = 1 A � 28 ∼ 2 . 15 13 21 1 . 62 4 which is as close as possible to 3 . 2 Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 16 / 21
Example 3 1 0 2 2 1 5 For A = , what are the least squares solutions to − 1 1 − 1 0 1 1 1 − 1 A x = b = ? − 1 2 6 1 13 0 A T A = A T b = 1 3 5 0 , . 13 5 31 0 Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 17 / 21
For this example, solving A T A ˆ x = A T b is equivalent to finding the null space of A T A 6 1 13 1 0 2 rref 1 3 5 − − → 0 1 1 13 5 31 0 0 0 Here, x 3 is free and x 2 = − x 3 , x 1 = − 2 x 3 . 2 So Nul A T A = R 1 . − 1 Here A ˆ x = 0 –not a very good approximation! Remember that we are looking for the vectors that map to the closest point to b in Col A . Dr Scott Morrison (ANU) MATH1014 Notes Second Semester 2015 18 / 21
Recommend
More recommend
