Section 5 Approximation Theory Numerical Analysis I Xiaojing Ye, - PowerPoint PPT Presentation

Section 5 Approximation Theory Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 183

Least squares approximation Given N data points { ( x i , y i ) } for i = 1 , . . . , N , can we determine a linear model y = a 1 x + a 0 (i.e., find a 0 , a 1 ) that fits the data? Table 8.1 y 16 x i y i x i y i 14 12 1 1.3 6 8.8 10 2 3.5 7 10.1 8 3 4.2 8 12.5 6 4 5.0 9 13.0 4 2 5 7.0 10 15.6 x 2 4 6 8 10 Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 184

Matrix formulation We can simplify notations by using matrices and vectors:     1 y 1 x 1 1 y 2 x 2     ∈ R N × 2 ∈ R N ,     y = X = . . .     . . . . . .         1 y N x N So we want to find a = ( a 0 , a 1 ) ⊤ ∈ R 2 such that y ≈ Xa . Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 185

Several types of fitting criteria There are several types of criteria for “best fitting”: ◮ Define the error function as E ∞ ( a ) = � y − Xa � ∞ and find a ∗ = arg min a E ∞ ( a ). This is also called the minimax problem since the problem min a E ∞ ( a ) can be written as min 1 ≤ i ≤ n | y i − ( a 0 + a 1 x i ) | max a ◮ Define the error function as E 1 ( a ) = � y − Xa � 1 and find a ∗ = arg min a E 1 ( a ). E 1 is also called the absolute deviation . Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 186

Least squares fitting In this course, we focus on the widely used least squares . Define the least squares error function as n � | y i − ( a 0 + a 1 x i ) | 2 E 2 ( a ) = � y − Xa � 2 = i =1 and the least squares solution a ∗ is a ∗ = arg min a E 2 ( a ) Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 187

Least squares fitting To find the optimal parameter a , we need to solve ∇ E 2 ( a ) = 2 X ⊤ ( Xa − y ) = 0 This is equivalent to the so-called normal equation : X ⊤ Xa = X ⊤ y Note that X ⊤ X ∈ R 2 × 2 and X ⊤ y ∈ R 2 , so the normal equation is easy to solve! Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 188

Least squares fitting It is easy to show that � � � � N � � N N i =1 x i i =1 y i X ⊤ X = X ⊤ y = , � N � N � N i =1 x 2 i =1 x i i =1 x i y i i Using the close-form of inverse of 2-by-2 matrix, we have � � N � − � N i =1 x 2 1 i =1 x i ( X ⊤ X ) − 1 = i − � N N � N i − ( � N i =1 x 2 i =1 x i ) 2 i =1 x i N Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 189

Least squares fitting Therefore we have the solution � � a 0 a ∗ = = ( X ⊤ X ) − 1 ( X ⊤ y ) a 1   � N � N i =1 y i − � N � N i =1 x 2 i =1 x i y i i =1 x i i N � N i − ( � N i =1 x 2 i =1 x i ) 2 =   N � N i =1 x i y i − � N � N i =1 x i i =1 y i   N � N i − ( � N i =1 x 2 i =1 x i ) 2 Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 190

Least squares fitting Example Least squares fitting of the data gives a 0 = − 0 . 36 and a 1 = 1 . 538. Table 8.1 y x i y i x i y i 16 14 1 1.3 6 8.8 12 10 2 3.5 7 10.1 8 y � 1.538 x � 0.360 3 4.2 8 12.5 6 4 5.0 9 13.0 4 2 5 7.0 10 15.6 x 2 4 6 8 10 Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 191

Polynomial least squares The least squares fitting presented above is also called linear least squares due to the linear model y = a 0 + a 1 x . For general least squares fitting problems with data { ( x i , y i ) : i = 1 , . . . , N } , we may use polynomial P n ( x ) = a 0 + a 1 x + a 2 x 2 + · · · + a n x n as the fitting model. Note that n = 1 reduces to linear model. Now the polynomial least squares error is defined by N � | y i − P n ( x i ) | 2 E ( a ) = i =1 where a = ( a 0 , a 1 , . . . , a n ) ⊤ ∈ R n +1 . Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 192

Matrices in polynomial least squares fitting Like before, we use matrices and vectors:     x 2 x n 1 · · · y 1 x 1 1 1 x 2 x n 1 · · · y 2 x 2     2 2 ∈ R N × ( n +1) ∈ R N ,     y = X = . . . . .     . . . . . . . . . .         x 2 x n 1 · · · y N x N N N So we want to find a = ( a 0 , a 1 , . . . , a n ) ⊤ ∈ R n +1 such that y ≈ Xa . Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 193

Polynomial least squares fitting Same as above, we need to find a such that ∇ E 2 ( a ) = 2 X ⊤ ( Xa − y ) = 0 which has normal equation : X ⊤ Xa = X ⊤ y Note that now X ⊤ X ∈ R ( n +1) × ( n +1) and X ⊤ y ∈ R n +1 . From normal equation we can solve for the fitting parameter   a 0 a 1   a ∗ = = ( X ⊤ X ) − 1 ( X ⊤ y )   .   . .     a n Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 194

Polynomial least squares Example Least squares fitting of the data using n = 2 gives a 0 = 1 . 0051 , a 1 = 0 . 86468 , a 2 = 0 . 84316. y i x i y i 2 1 0 1.0000 2 0.25 1.2840 y � 1.0051 � 0.86468 x � 0.84316 x 2 1 3 0.50 1.6487 4 0.75 2.1170 x 5 1.00 2.7183 0.25 0.50 0.75 1.00 Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 195

Other least squares fitting models In some situations, one may design model as y = be ax y = bx a as well as many others. To use least squares fitting, we note that they are equivalent to, respectively, log y = log b + ax log y = log b + a log x Therefore, we can first convert ( x i , y i ) to ( x i , log y i ) and (log x i , log y i ), and then apply standard linear least squares fitting. Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 196

Approximating functions We now consider fitting (approximation) of a given function f ( x ) ∈ C [ a , b ] Suppose we use a polynomial P n ( x ) of degree n to fit f ( x ), where P n ( x ) = a 0 + a 1 x + a 2 x 2 + · · · + a n x n with fitting parameters a = ( a 0 , a 1 , . . . , a n ) ⊤ ∈ R n +1 . Then the least squares error is 2 � b � b n | f ( x ) − P n ( x ) | 2 d x = � a k x k � � E ( a ) = � f ( x ) − d x � � � a a k =0 Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 197

Approximating functions The fitting parameter a needs to be solved from ∇ E ( a ) = 0. To this end, we first rewrite E ( a ) as a k x k � 2 � b � b � b n � n ( f ( x )) 2 d x − 2 � � x k f ( x ) d x + E ( a ) = d x a k a a a k =0 k =0 ∂ a n ) ⊤ ∈ R n +1 where Therefore ∇ E ( a ) = ( ∂ E ∂ a 0 , ∂ E ∂ a 1 , . . . , ∂ E � b � b n ∂ E x j + k d x � x j f ( x ) d x + 2 = − 2 a k ∂ a j a a k =0 for j = 0 , 1 , . . . , n . Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 198

Approximating functions By setting ∂ E ∂ a j = 0 for all j , we obtain the normal equation �� b � b n x j + k d x � � x j f ( x ) d x a k = a a k =0 for j = 0 , . . . , n . This is a linear system of n + 1 equations, from which we can solve for a ∗ = ( a 0 , . . . , a n ) ⊤ . Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 199

Approximating functions For the given function f ( x ) ∈ C [ a , b ], we obtain least squares approximating polynomial P n ( x ): y f ( x ) n � a k x k P n ( x ) � n 2 ( ( k � 0 � a k x k f ( x ) � k � 0 x a b Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 200

Approximating functions Example Use least squares approximating polynomial of degree 2 for the function f ( x ) = sin( π x ) on the interval [0 , 1]. y y � sin π x 1.0 0.8 0.6 y = P 2 ( x ) 0.4 0.2 x 0.2 0.4 0.6 0.8 1.0 Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 201

Least squares approximations with polynomials Remark ◮ The matrix in the normal equation is called Hilbert matrix , with entries of form � b x j + k d x = b j + k +1 − a j + k +1 j + k + 1 a which is prune to round-off errors. ◮ The parameters a = ( a 0 , . . . , a n ) ⊤ we obtained for polynomial P n ( x ) cannot be used for P n +1 ( x ) – we need to start the computations from beginning. Numerical Analysis I – Xiaojing Ye, Math & Stat, Georgia State University 202