Function Approximation Wouter J. Den Haan London School of - PowerPoint PPT Presentation

Function Approximation Wouter J. Den Haan London School of Economics � by Wouter J. Den Haan c August 18, 2015

Overview Polynomial approximations Splines Extra Goal Obtain an approximation for f ( x ) when • f ( x ) is unknown, but we have some information, or • f ( x ) is known, but too complex to work with

Overview Polynomial approximations Splines Extra Information available • Either finite set of derivatives • usually at one point • or finite set of function values • f 1 , · · · , f m at m nodes, x 1 , · · · , x m

Overview Polynomial approximations Splines Extra Classes of approximating functions 1 polynomials • this still gives lots of flexibility • examples of second-order polynomials • a 0 + a 1 x + a 2 x 2 • a 0 + a 1 ln ( x ) + a 2 ( ln ( x )) 2 � a 0 + a 1 ln ( x ) + a 2 ( ln ( x )) 2 � • exp 2 splines, e.g., linear interpolation

Overview Polynomial approximations Splines Extra Classes of approximating functions • Polynomials and splines can be expressed as n ∑ f ( x ) ≈ α i T i ( x ) i = 0 • T i ( x ) : the basis functions that define the class of functions used, e.g., for regular polynomials: T i ( x ) = x i . • α i : the coefficients that pin down the particular approximation

Overview Polynomial approximations Splines Extra Reducing the dimensionality unknown f ( x ) : infinite dimensional object ∑ n i = 0 α i T i ( x ) : n + 1 elements

Overview Polynomial approximations Splines Extra General procedure • Fix the order of the approximation n • Find the coefficients α 0 , · · · , α n • Evaluate the approximation • If necessary, increase n to get a better approximation

Overview Polynomial approximations Splines Extra Weierstrass (sloppy definition but true) Let f : [ a , b ] − → R be any real-valued function. For large enough n , it is approximated arbitrarily well with the polynomial n α i x i . ∑ i = 0 Thus, we can get an accurate approximation if • f is not a polynomial • f is discontinuous How can this be true?

Overview Polynomial approximations Splines Extra How to find the coefficients of the approximating polynomial? • With derivatives: • use the Taylor expansion • With a set of points (nodes), x 0 , · · · , x m , and function values, f 0 , · · · , f m ? • use projection • Lagrange way of writing the polynomial (see last part of slides)

Overview Polynomial approximations Splines Extra Function fitting as a projection Let     T 0 ( x 0 ) T 1 ( x 0 ) · · · T n ( x 0 ) f 0   T 0 ( x 1 ) T 1 ( x 1 ) · · · T n ( x 1 )     . . Y =  , X =    . . . . ... . . .   . . . f m T 0 ( x m ) T 1 ( x m ) · · · T n ( x m ) then Y ≈ X α • We need m ≥ n + 1 . Is m = n + 1 as bad as it is in empirical work? • What problem do you run into if n increases?

Overview Polynomial approximations Splines Extra Orthogonal polynomials • Construct basis functions so that they are orthogonal to each other, i.e., � b a T i ( x ) T j ( x ) w ( x ) dx = 0 ∀ i , j � i � = j • This requires a particular weighting function (density), w ( x ) , and range on which variables are defined, [ a , b ]

Overview Polynomial approximations Splines Extra Chebyshev orthogonal polynomials • 1 [ a , b ] = [ − 1, 1 ] and w ( x ) = ( 1 − x 2 ) 1/2 • What if function of interest is not defined on [ − 1, 1 ] ?

Overview Polynomial approximations Splines Extra Constructing Chebyshev polynomials • The basis functions of the Chebyshev polynomials are given by T c 0 ( x ) = 1 T c 1 ( x ) = x T c 2 xT c i ( x ) − T c i + 1 ( x ) = i − 1 ( x ) i > 1

Overview Polynomial approximations Splines Extra Chebyshev versus regular polynomials • Chebyshev polynomials, i.e., n a j T c ∑ f ( x ) ≈ j ( x ) , j = 0 can be rewritten as regular polynomials, i.e., n b j x j , ∑ f ( x ) ≈ j = 0

Overview Polynomial approximations Splines Extra Chebyshev nodes • The n th − order Chebyshev basis function has n solutions to T c n ( x ) = 0 • These are the n Chebyshev nodes

Overview Polynomial approximations Splines Extra Discrete orthogonality property • Evaluated at the Chebyshev nodes, the Chebyshev polynomials satisfy: n T c j ( x i ) T c ∑ k ( x i ) = 0 for j � = k i = 1 • Thus, if   T 0 ( x 0 ) T 1 ( x 0 ) · · · T n ( x 0 )   T 0 ( x 1 ) T 1 ( x 1 ) · · · T n ( x 1 )   X =   . . . ... . . .   . . . T 0 ( x m ) T 1 ( x m ) · · · T n ( x m ) then X � X is a diagonal matrix

Overview Polynomial approximations Splines Extra Uniform convergence • Weierstrass = ⇒ there is a good polynomial approximation • Weierstrass � f ( x ) = lim n → ∞ p n ( x ) for every sequence p n ( x ) • If polynomials are fitted on Chebyshev nodes = ⇒ even uniform convergence is guaranteed

Overview Polynomial approximations Splines Extra Splines Inputs: 1 n + 1 nodes, x 0 , · · · , x n 2 n + 1 function values, f ( x 0 ) · · · , f ( x n ) • nodes are fixed = ⇒ the n + 1 function values are the coefficients of the spline

Overview Polynomial approximations Splines Extra Piece-wise linear • For x ∈ [ x i , x i + 1 ] � � � x − x i � x − x i f ( x ) ≈ 1 − f i + f i + 1 . x i + 1 − x i x i + 1 − x i • That is, a separate linear function is fitted on the n intervals • Still it is easier/better to think of the coefficients of the approximating function as the n + 1 function values

Overview Polynomial approximations Splines Extra Piece-wise linear versus polynomial • Advantage: Shape preserving • in particular monotonicity & concavity (strict?) • Disadvantage: not differentiable

Overview Polynomial approximations Splines Extra Extra material 1 Lagrange interpolation 2 Higher dimensional polynomials 3 Higher-order splines

Overview Polynomial approximations Splines Extra Lagrange interpolation Let ( x − x 0 ) · · · ( x − x i − 1 )( x − x i + 1 ) · · · ( x − x n ) L i ( x ) = ( x i − x 0 ) · · · ( x i − x i − 1 )( x i − x i + 1 ) · · · ( x i − x n ) and f ( x ) ≈ f 0 L 0 ( x ) + · · · + f n L n ( x ) . • Right-hand side is an n th -order polynomial • By construction perfect fit at the n + 1 nodes? ⇒ the RHS is the n th -order approximation • =

Overview Polynomial approximations Splines Extra Higher-dimensional functions • second-order complete polynomial in x and y : a i , j x i y j ∑ 0 ≤ i + j ≤ 2 • second-order tensor product polynomial in x and y : 2 2 a i , j x i y j ∑ ∑ i = 0 j = 0

Overview Polynomial approximations Splines Extra Complete versus tensor product • tensor product can make programming easier • simple double loop instead of condition on sum • n th tensor has higher order term than ( n + 1 ) th complete • 2 nd -order tensor has fourth-order power • at least locally, lower-order powers are more important = ⇒ complete polynomial may be more efficient

Overview Polynomial approximations Splines Extra Higher-order spline Cubic (for example) • !!! Same inputs as with linear spline, i.e. n + 1 function values at n + 1 nodes which can still be thought of as the n + 1 coefficients that determine approximating function • Now fit 3 rd -order polynomials on each of the n intervals f ( x ) ≈ a i + b i x + c i x 2 + d i x 3 for x ∈ [ x i − 1 , x i ] . What conditions can we use to pin down these coefficients?

Overview Polynomial approximations Splines Extra Cubic spline conditions: levels • We have 2 + 2 ( n − 1 ) conditions to ensure that the function values correspond to the given function values at the nodes. • For the intermediate nodes we need that the cubic approximations of both adjacent segments give the correct answer. For example, we need that a 1 + b 1 x 1 + c 1 x 2 1 + d 1 x 3 f 1 = 1 and a 2 + b 2 x 1 + c 2 x 2 1 + d 2 x 3 f 1 = 1 • For the two endpoints, x 0 and x n + 1 , we only have one cubic that has to fit it correctly.

Overview Polynomial approximations Splines Extra Cubic spline conditions: 1 st -order derivatives • To ensure differentiability at the intermediate nodes we need b i + 2 c i x i + 3 d i x 2 i = b i + 1 + 2 c i + 1 x i + 3 d i + 1 x 2 i for x i ∈ { x 1 , · · · , x n − which gives us n − 1 conditions.

Overview Polynomial approximations Splines Extra Cubic spline conditions: 2 nd -order derivatives • To ensure that second derivatives are equal we need 2 c i + 6 d i x i = 2 c i + 1 + 6 d i + 1 x i for x i ∈ { x 1 , · · · , x n − 1 } . • We now have 2 + 4 ( n − 1 ) = 4 n − 2 conditions to find 4 n unknowns. • We need two additional conditions; e.g. that 2 nd -order derivatives at end points are zero.

Overview Polynomial approximations Splines Extra Splines - additional issues • (standard) higher-order splines do not preserve shape • higher-order difficult for multi-dimensional problems • first-order trivial for multi-dimensional problems • if interval is small then nondifferentiability often doesn’t matter