Regression, Curve Fitting and Optimisation Sam Tickle Supervised by - PowerPoint PPT Presentation

Introduction Nelder-Mead Algorithm Stochastic Algorithms A Non-Parametric Approach ‘Hard’ Functions An Application: Extreme Value Theory Conclusion Regression, Curve Fitting and Optimisation Sam Tickle Supervised by Elena Zanini STOR-i, University of Lancaster 4 September 2015 Sam Tickle Regression, Curve Fitting and Optimisation

Introduction Nelder-Mead Algorithm Stochastic Algorithms A Non-Parametric Approach ‘Hard’ Functions An Application: Extreme Value Theory Conclusion Introduction 1 Root Finding Nelder-Mead Algorithm 2 Stochastic Algorithms 3 Simulated Annealing A Non-Parametric Approach 4 ‘Hard’ Functions 5 The Rosenbrock Banana Function An Application: Extreme Value Theory 6 Conclusion 7 Sam Tickle Regression, Curve Fitting and Optimisation

Introduction Nelder-Mead Algorithm Stochastic Algorithms A Non-Parametric Approach ‘Hard’ Functions An Application: Extreme Value Theory Conclusion Given a set of data, what is the optimum curve which may be fitted? This question has obvious importance in queries regarding relationships between two or more variables, as well as explaining data quantitatively. Sam Tickle Regression, Curve Fitting and Optimisation

Introduction Nelder-Mead Algorithm Stochastic Algorithms A Non-Parametric Approach ‘Hard’ Functions An Application: Extreme Value Theory Conclusion If a straight line is needed, we can do the standard trick of using Ordinary Least Squares (OLS). However, there will be situations in which this may not be appropriate. Sam Tickle Regression, Curve Fitting and Optimisation

Introduction Nelder-Mead Algorithm Stochastic Algorithms A Non-Parametric Approach ‘Hard’ Functions An Application: Extreme Value Theory Conclusion Some Less Trivial Examples 30 400 25 2.0 300 20 1.5 y y y 15 200 1.0 10 100 5 0.5 0 0 5 10 15 20 0 5 10 15 0 10 20 30 x x x Sam Tickle Regression, Curve Fitting and Optimisation

Introduction Nelder-Mead Algorithm Stochastic Algorithms A Non-Parametric Approach ‘Hard’ Functions An Application: Extreme Value Theory Conclusion We observe that the OLS inference arises from an optimisation problem, namely argmin b ∈ R p || Y − Xb || 2 . So it makes sense to think about the problem of optimal curve fitting from the perspective of optimisation. Sam Tickle Regression, Curve Fitting and Optimisation

Introduction Nelder-Mead Algorithm Stochastic Algorithms A Non-Parametric Approach ‘Hard’ Functions An Application: Extreme Value Theory Conclusion Optimisation has an obvious analogue in root finding. There are several core methods we can use for this: Bisection; Newton-Raphson; Secant; Muller’s. All of these (except Newton-Raphson) are derivative-free . Sam Tickle Regression, Curve Fitting and Optimisation

Introduction Nelder-Mead Algorithm Stochastic Algorithms A Non-Parametric Approach ‘Hard’ Functions An Application: Extreme Value Theory Conclusion In higher dimensions, one of the more effective non-derivative-free methods is the Broydon-Fletcher-Goldfarb-Shanno (BFGS) Method , which can be adapted to optimise by changing the iterative equation to x n +1 = x n − [ Hf ( x n )] − 1 ∇ f ( x n ). Sam Tickle Regression, Curve Fitting and Optimisation

Introduction Nelder-Mead Algorithm Stochastic Algorithms A Non-Parametric Approach ‘Hard’ Functions An Application: Extreme Value Theory Conclusion The Nelder-Mead Algorithm Suppose our goal is to minimise the function f ( x ), where x ∈ R n . Sam Tickle Regression, Curve Fitting and Optimisation

Introduction Nelder-Mead Algorithm Stochastic Algorithms A Non-Parametric Approach ‘Hard’ Functions An Application: Extreme Value Theory Conclusion Start with n + 1 test points: x 1 , ..., x n +1 . Sam Tickle Regression, Curve Fitting and Optimisation

Introduction Nelder-Mead Algorithm Stochastic Algorithms A Non-Parametric Approach ‘Hard’ Functions An Application: Extreme Value Theory Conclusion Order these points by output value, so that f ( x 1 ) ≤ f ( x 2 ) ≤ ... ≤ f ( x n +1 ). x 3 x 1 x 2 Sam Tickle Regression, Curve Fitting and Optimisation

Introduction Nelder-Mead Algorithm Stochastic Algorithms A Non-Parametric Approach ‘Hard’ Functions An Application: Extreme Value Theory Conclusion We consider several different ‘candidate points’, and if these aren’t an improvement, then we shrink the simplex. Sam Tickle Regression, Curve Fitting and Optimisation

Introduction Nelder-Mead Algorithm Stochastic Algorithms A Non-Parametric Approach ‘Hard’ Functions An Application: Extreme Value Theory Conclusion How well does this work on the problem? 400 15 2.0 300 1.5 10 yrange yrange yrange 200 1.0 5 100 0.5 0 5 10 15 20 0 5 10 15 5 10 15 20 25 xrange xrange xrange Sam Tickle Regression, Curve Fitting and Optimisation

Introduction Nelder-Mead Algorithm Stochastic Algorithms A Non-Parametric Approach ‘Hard’ Functions An Application: Extreme Value Theory Conclusion Disadvantages of Nelder-Mead We usually require a reasonable idea of the form of the relationship between the two variables in question to produce a reasonable eventual plot; If the data do not conform well to the ‘true’ underlying relationship, the procedure can be very costly , and could arrive at an incorrect answer if the initial conditions are poorly specified. Sam Tickle Regression, Curve Fitting and Optimisation

Introduction Nelder-Mead Algorithm Stochastic Algorithms A Non-Parametric Approach ‘Hard’ Functions An Application: Extreme Value Theory Conclusion Stochastic Algorithms Several alternative methods of optimisation can be used which employ a probabilistic approach. These include: Simulated Annealing; Genetic Algorithms; Ant Colony Optimisation. Sam Tickle Regression, Curve Fitting and Optimisation

Introduction Nelder-Mead Algorithm Stochastic Algorithms A Non-Parametric Approach ‘Hard’ Functions An Application: Extreme Value Theory Conclusion Simulated Annealing (SA) is a physical process describing the cooling of a material in a system with a controlled negative temperature gradient. It can be observed that under situations where a substance such as water cools in such a system, an ‘optimal’ solid arrangement is obtained. Sam Tickle Regression, Curve Fitting and Optimisation

Introduction Nelder-Mead Algorithm Stochastic Algorithms A Non-Parametric Approach ‘Hard’ Functions An Application: Extreme Value Theory Conclusion How SA works To use Simulated Annealing in an optimisation problem, the following need to be well defined: The neighbours of each state - e.g. for a discrete domain, a rearrangement of two adjacent states; The energies of each state; The probability of moving from state S to state S ′ - states with smaller energy preferred, so P ( E , E ′ , T ) > P ( E , E ′′ , T ) when E ′ < E ′′ . Sam Tickle Regression, Curve Fitting and Optimisation

Introduction Nelder-Mead Algorithm Stochastic Algorithms A Non-Parametric Approach ‘Hard’ Functions An Application: Extreme Value Theory Conclusion How SA works In the problem of curve fitting: We shall define a ‘neighbour’ of the current curve as an addition of a small, simple function; The probabilites shall be set as follows: If E < E ′ , then P ( E , E ′ , T ) ∝ exp ( E − E ′ ); T Else, P ( E , E ′ , T ) ∝ 1. Sam Tickle Regression, Curve Fitting and Optimisation

Introduction Nelder-Mead Algorithm Stochastic Algorithms A Non-Parametric Approach ‘Hard’ Functions An Application: Extreme Value Theory Conclusion How well does this work on the problem? 18 14 16 12 14 10 8 12 yrange yrange 6 10 4 8 2 6 0 0 10 20 30 5 10 15 20 25 xrange xrange Sam Tickle Regression, Curve Fitting and Optimisation

Introduction Nelder-Mead Algorithm Stochastic Algorithms A Non-Parametric Approach ‘Hard’ Functions An Application: Extreme Value Theory Conclusion Disadvantages of SA Often requires a high starting temperature to achieve a reasonable result; The model is very sensitive to starting temperature - choice is not obvious; Is very difficult to achieve a fairly accuracte solution, as it is difficult to construct well defined neighbours which enable effective ‘zeroing in’ on a state in a continuous domain. Sam Tickle Regression, Curve Fitting and Optimisation

Introduction Nelder-Mead Algorithm Stochastic Algorithms A Non-Parametric Approach ‘Hard’ Functions An Application: Extreme Value Theory Conclusion A Non-Parametric Approach Suppose we had no intuition at all as to an underlying relationship, such as in the example shown below. 30 25 20 y 15 10 5 0 0 10 20 30 x Sam Tickle Regression, Curve Fitting and Optimisation

Introduction Nelder-Mead Algorithm Stochastic Algorithms A Non-Parametric Approach ‘Hard’ Functions An Application: Extreme Value Theory Conclusion One way of tackling the problem of curve fitting in this instance is to give each point an associated ‘reward’ function, with shape similar to a hillock. Sam Tickle Regression, Curve Fitting and Optimisation