Efficiency of Gaussian and Cauchy functions in Function method the - PowerPoint PPT Presentation

Efficiency of Gaussian and Cauchy functions in the Filled Efficiency of Gaussian and Cauchy functions in Function method the Filled Function method Jos´ e Guadalu- pe Flores Mu˜ niz, Vyacheslav V. Kalashnikov, Nataliya Jos´ e Guadalupe Flores Mu˜ niz, Kalashnykova, and Vladik Vyacheslav V. Kalashnikov, Kreinovich Nataliya Kalashnykova, Outline and Vladik Kreinovich Formulation of the Problem Need for Smoothing 27 de octubre de 2016 Need to Select an Appropriate Value σ We May Need Several

Outline I Efficiency of Gaussian and Cauchy functions in the Filled Function One of the main problems of optimization algorithms is method that they end up in a local optimum. Jos´ e Guadalu- pe Flores It is necessary to get out of the local optimum and Mu˜ niz, Vyacheslav V. eventually reach the global optimum. Kalashnikov, Nataliya Kalashnykova, One of the promising methods to leave the local optimum and Vladik is the filled function method. Kreinovich Empirically, the best smoothing functions in this method Outline Formulation of are the Gaussian and the Cauchy functions. the Problem In this talk, we provide a possible theoretical explanation Need for Smoothing for this empirical result. Need to Select an Appropriate Value σ We May Need Several

Formulation of the Problem I Efficiency of Gaussian and Cauchy functions in the Filled Function method In the Renpu’s filled function method, once we reach a Jos´ e Guadalu- local optimum x ∗ , we optimize an auxiliary expression pe Flores Mu˜ niz, Vyacheslav V. � x − x ∗ � Kalashnikov, · F ( f ( x ) , f ( x ∗ ) , x ) + G ( f ( x ) , f ( x ∗ ) , x ) , Nataliya K Kalashnykova, σ and Vladik Kreinovich for some K, F, G, and σ. Outline Formulation of We use its optimum as a new first approximation to find the Problem the optimum of f ( x ) . Need for Smoothing Need to Select an Appropriate Value σ We May Need Several

Formulation of the Problem II Efficiency of Gaussian and Cauchy functions in the Filled Several different functions K ( x − x ∗ ) have been proposed, Function method but it turns out that the most computationally efficient Jos´ e Guadalu- pe Flores functions are the Gaussian and Cauchy functions Mu˜ niz, Vyacheslav V. Kalashnikov, 1 Nataliya K ( x ) = exp( −� x � 2 ) , K ( x ) = 1 + � x � 2 . Kalashnykova, and Vladik Kreinovich Are these function indeed the most efficient? or they are Outline simply the most efficient among a few functions that have Formulation of the Problem been tried? Need for Smoothing Need to Select an Appropriate Value σ We May Need Several

Need for Smoothing I Efficiency of Gaussian and Cauchy functions in the Filled Function method One of the known ways to eliminate local optima is to Jos´ e Guadalu- apply a weighted smoothing . pe Flores Mu˜ niz, Vyacheslav V. In this method, we replace the original objective function Kalashnikov, Nataliya f ( x ) with a “smoothed” one Kalashnykova, and Vladik Kreinovich � x − x ′ � � f ∗ ( x ) def · f ( x ′ ) dx ′ , = K Outline σ Formulation of the Problem for some K ( x ) and σ . Need for Smoothing Need to Select an Appropriate Value σ We May Need Several

Need for Smoothing II Efficiency of Gaussian and Cauchy functions in the Filled Function method The weighting function is usually selected in such a way Jos´ e Guadalu- � pe Flores that K ( − x ) = K ( x ) and K ( x ) dx < + ∞ . Mu˜ niz, Vyacheslav V. Kalashnikov, The first condition comes from the fact that we have no Nataliya Kalashnykova, reason to prefer different orientations of coordinates. and Vladik Kreinovich The second condition is that for f ( x ) = const , smoothing Outline should leads to a finite constant. Formulation of the Problem Need for Smoothing Need to Select an Appropriate Value σ We May Need Several

Need to Select an Appropriate Value σ I Efficiency of Gaussian and Cauchy functions in the Filled When σ is too small, the smoothing only covers a very Function method small neighborhood of each point x . Jos´ e Guadalu- The smoothed function f ∗ ( x ) is close to the original pe Flores Mu˜ niz, objective function f ( x ) . Vyacheslav V. Kalashnikov, Nataliya So, we will still observe all the local optima. Kalashnykova, and Vladik On the other hand, if σ is too large, the smoothed Kreinovich function f ∗ ( x ) is too different from f ( x ) . Outline So the optimum of the smoothed function may have Formulation of the Problem nothing to do with the optimum of f ( x ) . Need for Smoothing So, for the smoothing method to work, it is important to Need to Select select an appropriate value of σ . an Appropriate Value σ We May Need Several

We May Need Several Iterations to Find an Appropriate σ I Efficiency of Gaussian and Cauchy functions in the Filled Function method Jos´ e Guadalu- pe Flores Our first estimate for σ may not be the best. Mu˜ niz, Vyacheslav V. Kalashnikov, If we have smoothed the function too much, then we need Nataliya Kalashnykova, to “un-smooth” it, i.e., to select a smaller σ . and Vladik Kreinovich If we have not smoothed the function enough, then we need to smooth it more, i.e., to select a larger σ . Outline Formulation of the Problem Need for Smoothing Need to Select an Appropriate Value σ We May Need Several

Computationally Efficient Smoothing: Analysis I Efficiency of Gaussian and Cauchy functions in Once we have smoothed the function too much, it is the Filled Function difficult to un-smooth it, therefore, a usual approach is method Jos´ e Guadalu- that we first try some small smoothing. pe Flores Mu˜ niz, If the resulting smoothed function f ∗ ( x ) still leads to a Vyacheslav V. Kalashnikov, similar local maximum, we smooth it some more, etc. Nataliya Kalashnykova, For small σ : and Vladik Kreinovich to find each value f ∗ ( x ) of the smoothed function, we only need to consider values of f ( x ′ ) in a small vicinity Outline of x . Formulation of the Problem The larger σ , the larger this vicinity, so: Need for Smoothing the more values f ( x ′ ) we need to take into account, Need to Select and thus the more computations we need. an Appropriate Value σ We May Need Several

Computationally Efficient Smoothing: Conclusion I Efficiency of Gaussian and Cauchy functions in the Filled Function method Let’s assume that we have a smoothed function f ∗ ( x ) Jos´ e Guadalu- corresponding to some value of σ . pe Flores Mu˜ niz, Vyacheslav V. We need to compute a smoothed function f ∗∗ ( x ) Kalashnikov, corresponding to a larger value σ ′ > σ . Nataliya Kalashnykova, and Vladik It is thus more computationally efficient not to apply Kreinovich smoothing with σ ′ to the original f ( x ) . Outline Instead, we should apply a small additional smoothing to Formulation of the Problem the smoothed function f ∗ ( x ) . Need for Smoothing Need to Select an Appropriate Value σ We May Need Several

Resulting Requirement on the Smoothing Function K ( x ) I Efficiency of For every σ ′ and σ , there should be an appropriate value Gaussian and Cauchy functions in ∆ σ . the Filled Function Then, after we get method Jos´ e Guadalu- � x − x ′ � � pe Flores f ∗ ( x ) = · f ( x ′ ) dx ′ , K Mu˜ niz, σ Vyacheslav V. Kalashnikov, Nataliya a smoothing with ∆ σ should lead to the desired function Kalashnykova, and Vladik Kreinovich � x − x ′ � � f ∗∗ ( x ) = · f ( x ′ ) dx ′ . K σ ′ Outline Formulation of In other words, we need to make sure that for every the Problem Need for objective function f ( x ) , we have Smoothing � x − x ′ � x − x ′ Need to Select � � � � · f ( x ′ ) dx ′ = · f ∗ ( x ′ ) dx ′ . an Appropriate K K Value σ σ ′ ∆ σ We May Need Several

Analyzing The Above Requirement I Efficiency of Gaussian and Cauchy functions in the Filled The above requirement leads to: Function method � x ′′ − x ′ � x − x ′ � x − x ′′ � � � � Jos´ e Guadalu- dx ′′ . = · K K K pe Flores σ ′ ∆ σ σ Mu˜ niz, Vyacheslav V. Kalashnikov, Nataliya The function K ( x ) is non-negative, and its integral Kalashnykova, � and Vladik K ( x ) dx is finite, thus, after dividing K ( x ) by the value Kreinovich of this integral, we get a probability density function (pdf): Outline Formulation of the Problem K ( x ) ρ X ( x ) = . Need for � Smoothing K ( y ) dy Need to Select an Appropriate Value σ We May Need Several

Analyzing The Above Requirement II Efficiency of Gaussian and Cauchy For this pdf: functions in the Filled Function � x ′′ − x ′ � x − x ′ � x − x ′′ � � � � method dx ′′ . ρ = ρ · ρ Jos´ e Guadalu- σ ′ ∆ σ σ pe Flores Mu˜ niz, Vyacheslav V. Let X denote the random variable with the probability Kalashnikov, Nataliya density function ρ X ( x ) . Kalashnykova, and Vladik Then, the LHS is pdf of σ ′ · X . Kreinovich The RHS is a pdf of the sum of two independent random Outline Formulation of variables ∼ σ · X and ∼ ∆ σ · X . the Problem The requirement that the sum is similarly distributed Need for Smoothing means that ρ ( x ) is infinitely divisible . Need to Select an Appropriate Value σ We May Need Several