Graphical Tools for the Analysis of Bi-objective Optimization - PowerPoint PPT Presentation

Graphical Tools for the Analysis of Bi-objective Optimization Algorithms nez 1 ıs Paquete 2 utzle 1 Manuel L´ opez-Ib´ a˜ Lu´ Thomas St¨ 1 manuel.lopez-ibanez @ ulb.ac.be 2 paquete @ dei.uc.pt stuetzle @ ulb.ac.be CISUC, Department of Informatics Engineering IRIDIA, CoDE, University of Coimbra, Portugal Universit´ e Libre de Bruxelles (ULB) Brussels, Belgium Workshop on Theoretical Aspects of Evolutionary Multiobjective Optimization GECCO, July 8, 2010 IRIDIA Institut de Recherches Interdisciplinaires et de Développements en Intelligence Artificielle

Is a multi-objective optimization algorithm better than another? Manuel L´ opez-Ib´ a˜ nez, Lu´ ıs Paquete, Thomas St¨ utzle EAF Graphical Tools for Bi-objective Optimization

Analysis of Multi-objective Optimization Algorithms Best criterion: dominance criteria among output sets ( ⊳ ) ✔ If A r ⊳ B r for all r runs, then A is better than B ✘ Output sets of high-performing algorithms are often incomparable in terms of dominance Unary/binary measures: ✔ Experimental analysis like in single-objective optimization ✔ Intuitively describe desirable properties ✘ Bias ✘ Loss of information (Over-simplification) Many unary/binary measures: ✔ Less bias ✔ Less information lost ✘ Difficult interpretation ✘ Consensus issues [Mersmann et al., 2010] Manuel L´ opez-Ib´ a˜ nez, Lu´ ıs Paquete, Thomas St¨ utzle EAF Graphical Tools for Bi-objective Optimization

Analysis of Multi-objective Optimization Algorithms In which aspect is a multi-objective optimization algorithm better than another? Manuel L´ opez-Ib´ a˜ nez, Lu´ ıs Paquete, Thomas St¨ utzle EAF Graphical Tools for Bi-objective Optimization

The Attainment Function [Grunert da Fonseca et al., 2001] Extends scalar concepts of location (mean, median) and spread (variance, IQR) to random sets Completely characterizes the statistical distribution of the output of multi-objective optimizers in terms of location, spread and mutual dependence [Fonseca et al., 2005] Enables statistical testing and inference [Fonseca et al., 2005; Grunert da Fonseca & Fonseca, 2010; Paquete & St¨ utzle, 2006, 2009] Theory more advanced than practical applications Manuel L´ opez-Ib´ a˜ nez, Lu´ ıs Paquete, Thomas St¨ utzle EAF Graphical Tools for Bi-objective Optimization

The Attainment Function [Grunert da Fonseca et al., 2001] First-order attainment function: v ) : R d → [ 0, 1 ] α ( � Probability of a random set attaining a particular point � v in the objective space An attainment function can characterize the output of a stochastic multi-objective optimization algorithm The real attainment function is unknown but. . . We can estimate it: Empirical attainment function (EAF) Manuel L´ opez-Ib´ a˜ nez, Lu´ ıs Paquete, Thomas St¨ utzle EAF Graphical Tools for Bi-objective Optimization

The Empirical Attainment Function f (x) f (x) f (x) f (x) f (x) f (x) 2 2 2 2 2 2 Run 1 Run 1 Run 1 Run 1 Run 1 Run 1 Run 2 Run 2 Run 2 Run 2 Run 3 Run 3 f (x) f (x) f (x) f (x) f (x) f (x) 1 1 1 1 1 1 Manuel L´ opez-Ib´ a˜ nez, Lu´ ıs Paquete, Thomas St¨ utzle EAF Graphical Tools for Bi-objective Optimization

Attainment Surfaces k % attainment surface: “ Lower boundary of the region in the objective space with a value of the attainment function of at least k / 100 . ” Empirical k % attainment surface: “ The line delimiting the objective space attained by at least k % of the runs of a multi-objective algorithm. ” Median attainment surface = 50% attainment surface Worst attainment surface = 100% attainment surface Best attainment surface = region attained by at least one run Manuel L´ opez-Ib´ a˜ nez, Lu´ ıs Paquete, Thomas St¨ utzle EAF Graphical Tools for Bi-objective Optimization

Attainment Surfaces 10 independent runs Attainment surfaces 4e+05 best 4e+05 median worst objective 2 objective 2 3.6e+05 3.6e+05 3.2e+05 3.2e+05 3.1e+05 3.4e+05 3.7e+05 4e+05 3.1e+05 3.4e+05 3.7e+05 4e+05 objective 1 objective 1 What is the “typical” behaviour? Less clutter, more information! Manuel L´ opez-Ib´ a˜ nez, Lu´ ıs Paquete, Thomas St¨ utzle EAF Graphical Tools for Bi-objective Optimization

Comparing Two Algorithms: EAFs side-by-side objective 1 2.8e+05 3.2e+05 3.6e+05 4e+05 4.4e+05 4e+05 4e+05 objective 2 objective 2 3.5e+05 3.5e+05 [0.8, 1.0] [0.6, 0.8) 3e+05 3e+05 [0.4, 0.6) [0.2, 0.4) [0.0, 0.2) 2.8e+05 3.2e+05 3.6e+05 4e+05 4.4e+05 objective 1 Algorithm 1 Algorithm 2 Manuel L´ opez-Ib´ a˜ nez, Lu´ ıs Paquete, Thomas St¨ utzle EAF Graphical Tools for Bi-objective Optimization

Comparing Two Algorithms: EAF Differences objective 1 2.8e+05 3.2e+05 3.6e+05 4e+05 4.4e+05 [0.8, 1.0] [0.6, 0.8) [0.4, 0.6) [0.2, 0.4) [0.0, 0.2) 4e+05 4e+05 objective 2 objective 2 3.5e+05 3.5e+05 3e+05 3e+05 2.8e+05 3.2e+05 3.6e+05 4e+05 4.4e+05 objective 1 Algorithm 1 Algorithm 2 Manuel L´ opez-Ib´ a˜ nez, Lu´ ıs Paquete, Thomas St¨ utzle EAF Graphical Tools for Bi-objective Optimization

EAF Differences: More examples [L´ opez-Ib´ a˜ nez & St¨ utzle, 2010b] Pareto ACO Manuel L´ opez-Ib´ a˜ nez, Lu´ ıs Paquete, Thomas St¨ utzle EAF Graphical Tools for Bi-objective Optimization

EAF Differences: More examples Permutation Flow-shop, Hybrid TP + PLS against MOSA [Dubois-Lacoste et al., 2010] C max 1.14e+04 1.16e+04 1.18e+04 1.2e+04 1.22e+04 [0.8, 1.0] [0.6, 0.8) [0.4, 0.6) 1.34e+06 [0.2, 0.4) [0.0, 0.2) ∑ C i 1.3e+06 1.26e+06 1.14e+04 1.16e+04 1.18e+04 1.2e+04 1.22e+04 C max TP+PLS MOSA Manuel L´ opez-Ib´ a˜ nez, Lu´ ıs Paquete, Thomas St¨ utzle EAF Graphical Tools for Bi-objective Optimization

Conclusions Not a replacement of dominance or quality measures, but a powerful exploratory data analysis tool Related earlier works by Knowles [2005] and Fonseca et al. [2005] Ongoing work on both theory and practical applications We make available software tools to produce these plots [L´ opez-Ib´ a˜ nez et al., 2010] http://iridia.ulb.ac.be/~manuel/eaftools Manuel L´ opez-Ib´ a˜ nez, Lu´ ıs Paquete, Thomas St¨ utzle EAF Graphical Tools for Bi-objective Optimization

Open Questions for Future Research 1 EAF for more than 2 dimensions No algorithm publicly available (ongoing work) Best way to use the EAF: direct visualization (at most 3D), parallel coordinates, . . . 2 How to summarise the results on several instances? 3 Practical applications of higher-order EAFs 4 Theoretical and practical challenges Manuel L´ opez-Ib´ a˜ nez, Lu´ ıs Paquete, Thomas St¨ utzle EAF Graphical Tools for Bi-objective Optimization

References I J. Dubois-Lacoste, M. L´ opez-Ib´ a˜ nez, and T. St¨ utzle. Supplementary material: A Hybrid TPLS+PLS Algorithm for Bi-objective Flow-shop Scheduling Problems. http://iridia.ulb.ac.be/supp/IridiaSupp2010-001 , 2010. C. M. Fonseca, V. Grunert da Fonseca, and L. Paquete. Exploring the performance of stochastic multiobjective optimisers with the second-order attainment function. In C. C. Coello, A. H. Aguirre, and E. Zitzler, editors, Evolutionary Multi-criterion Optimization (EMO 2005) , volume 3410 of Lecture Notes in Computer Science , pages 250–264. Springer, Heidelberg, Germany, 2005. V. Grunert da Fonseca and C. M. Fonseca. The attainment-function approach to stochastic multiobjective optimizer assessment and comparison. In T. Bartz-Beielstein, M. Chiarandini, L. Paquete, and M. Preuß, editors, Experimental Methods for the Analysis of Optimization Algorithms . Springer, 2010. V. Grunert da Fonseca, C. M. Fonseca, and A. O. Hall. Inferential performance assessment of stochastic optimisers and the attainment function. In E. Zitzler, K. Deb, L. Thiele, C. A. Coello, and D. Corne, editors, Proceedings of EMO 2001 , volume 1993 of Lecture Notes in Computer Science , pages 213–225. Springer, Heidelberg, Germany, 2001. Manuel L´ opez-Ib´ a˜ nez, Lu´ ıs Paquete, Thomas St¨ utzle EAF Graphical Tools for Bi-objective Optimization