Outline Heuristics for Continuos Optimization DM812 METAHEURISTICS - PowerPoint PPT Presentation

An Example Outline Heuristics for Continuos Optimization DM812 METAHEURISTICS Lecture 13 Heuristic Methods for Continuous 1. An Application Example in Econometrics Optimization 2. Heuristics for Continuos Optimization Marco Chiarandini Department of Mathematics and Computer Science University of Southern Denmark, Odense, Denmark <marco@imada.sdu.dk> An Example An Example Outline Capital Asset Pricing Model Heuristics for Continuos Optimization Heuristics for Continuos Optimization Tool for pricing an individual asset i Individual security’s Market’s securities = β i · reward-to-risk ratio reward-to-risk ratio � � � � E ( R i ) − R f = β i · E ( R m ) − R f 1. An Application Example in Econometrics β i sensitivity of the asset returns to market returns Under normality assumption and least squares method: 2. Heuristics for Continuos Optimization β i = Cov( R i , R m ) Var( R m ) Alternatively: R it − R ft = β 0 + β 1 · ( R mt − R ft ) Use more robust techniques than least squares to determine β 0 and β 1 [Winker, Lyra, Sharpe, 2008]

An Example An Example Least Median of Squares Heuristics for Continuos Optimization Heuristics for Continuos Optimization Y t = β 0 + β 1 X t + ǫ t 0.010 0.00020 � 2 ǫ 2 0.008 � t = Y t − β 0 − β 1 X t 0.00015 0.006 0.004 least squares method: 0.00010 n 0.002 � ǫ 2 min t 2.0 2.0 t =1 1.5 1.8 1.0 least median of squares method: 1.6 0.10 0.010 0.5 0.05 1.4 0.005 0.0 beta beta 0.00 0.000 1.2 −0.5 � � ǫ 2 �� min median −0.05 −0.005 t An Example An Example Outline Heuristics for Continuos Optimization Heuristics for Continuos Optimization Four solutions corresponding to four different local optima (red line: least squares; blue line: least median of squares) median( ε t 2 ) = 5.2e−05 median( ε t 2 ) = 0.00014 0.06 + 0.06 + + + + + + + + + + + + + + + + + + + + + 0.02 + + 0.02 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 1. An Application Example in Econometrics + + + + + + + + + + + + + + + + + + + + + + + ++ + + + + + + + + ++ + Y t + + + + + + + + + + Y t + + + + + + + + + + + ++ + + ++ + + + ++ + + ++ + + + + + + + + + + + + + + ++ + + + + + + + + + + + + + + + + + + + + + + + + + + + ++ + + + + + + + + + + + + + + + + + + + + + + + ++ + + + + + + + + ++ + + + + + + + + + + + + + + + + + + + + + + + + + ++ + + + + + + + + + + + + + + ++ + + + + + + + + + + + + + + + + + + + + + + + + + + + ++ + + + + + + + + ++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + −0.04 −0.04 + + + + + + + + + + + + + + + + + + + + + + + + −0.02 0.00 0.02 0.04 −0.02 0.00 0.02 0.04 2. Heuristics for Continuos Optimization X X median( ε t 2 ) = 8.6e−05 median( ε t 2 ) = 6.9e−05 0.06 + 0.06 + + + + + + + + + + + + + + + + + + + + + 0.02 0.02 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ++ + + + + + + ++ Y t + + + + + + + + Y t + + + + + + + + + + ++ + + + + + ++ + + + + + ++ + + + + + ++ + + + + + + + + ++ + + + + + + + ++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ++ + + + + + + + + + + + + ++ + + + + + + + + + + + + ++ + + + + + + + + ++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ++ + + + + + + + ++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + −0.04 −0.04 + + + + + + + + + + + + + + + + + + + + + + + + −0.02 0.00 0.02 0.04 −0.02 0.00 0.02 0.04

An Example An Example Optimization Heuristics Nelder-Mead Heuristics for Continuos Optimization Heuristics for Continuos Optimization Simplex based method [Spendley et al. (1962)] Nelder-Mead Simulated Annealing Differential Evolution Particle Swarm Optimization Genetic Algorithm Ant Colony Optimization An Example An Example Nelder-Mead (cont.) Nelder-Mead (cont.) Heuristics for Continuos Optimization Heuristics for Continuos Optimization Nelder-Mead simplex method [Nelder and Mead, 1965] :

An Example An Example Nelder-Mead (cont.) Simulated Annealing Heuristics for Continuos Optimization Heuristics for Continuos Optimization Simulated Annealing (SA): determine initial candidate solution s set initial temperature T = T 0 Example: while termination condition is not satisfied do while keep T constant, that is, T max iterations not elapsed do probabilistically choose a neighbor s ′ of s using proposal mechanism accept s ′ as new search position with probability: ( if f ( s ′ ) ≤ f ( s ) 1 p ( T, s, s ′ ) := exp f ( s ) − f ( s ′ ) otherwise T update T according to annealing schedule Proposal mechanism The next candidate point is generated from a Gaussian Markov kernel with scale proportional to the actual temperature. An Example An Example Simulated Annealing Differential Evolution Heuristics for Continuos Optimization Heuristics for Continuos Optimization Annealing schedule logarithmic cooling schedule [Belisle (1992)] T 0 T = ln( ⌊ i − 1 I max ⌋ I max + e ) Differential Evolution (DE) determine initial population P while termination criterion is not satisfied do 1.0 10 for each solution x of P do 0.8 generate solution u from three solutions of P by mutation 8 Temperature 0.6 Cooling generate solution v from u by recombination with solution x 6 0.4 select between x and v solutions 4 0.2 2 0.0 −40 −20 0 20 40 0 200 400 600 800 1000 x x threshold accepting [Dueck and Scheuer (1990)] accept if ∆ < τ

An Example An Example Differential Evolution (cont.) Differential Evolution (cont.) Heuristics for Continuos Optimization Heuristics for Continuos Optimization Solution representation: x = ( x 1 , x 2 , . . . , x p ) Mutation: u = r 1 + F · ( r 2 − r 3 ) F ∈ [0 , 2] and ( r 1 , r 2 , r 3 ) ∈ P Recombination: � if p < CR or j = r u j v j = j = 1 , 2 , . . . , p x j otherwise [ http://www.icsi.berkeley.edu/~storn/code.html Selection: replace x with v if f ( v ) is better K. Price and R. Storn, 1995] An Example An Example Particle Swarm Optimization Particle Swarm Optimization Heuristics for Continuos Optimization Heuristics for Continuos Optimization Particle Swarm Optimization

An Example Generation of Initial Solutions Heuristics for Continuos Optimization Point generators: Left: Uniform random distribution (pseudo random number generator) Right: Quasi-Monte Carlo method: low discrepancy sequence generator [Bratley, Fox and Niederreiter, 1994] 1.0 1.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.8 ● 0.8 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.6 ● 0.6 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● β 1 β 1 ● ● β ● β ● ● ● ● ● ● ● ● ● ● ● ● 0.4 ● ● 0.4 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.2 0.2 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.0 ● ● 0.0 ● ● ● ● ● 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 (for other methods see spatial point process from spatial statistics )