with heuristiclab
play

with HeuristicLab An Open Source Optimization Environment for - PowerPoint PPT Presentation

Jul 30, 2023 •536 likes •1.77k views

Algorithm and Experiment Design with HeuristicLab An Open Source Optimization Environment for Research and Education S. Wagner, G. Kronberger Heuristic and Evolutionary Algorithms Laboratory (HEAL) School of Informatics/Communications/Media,

  1. Create Batch Runs and Experiments drag & drop here to add additional algorithms, batch runs, experiments, etc. HeuristicLab Tutorial http://dev.heuristiclab.com 26

  2. Clipboard • Store items – click on the buttons to add or remove items – drag & drop items on the clipboard – use the menu to add a copy of a shown item to the clipboard • Show items – double-click on an item in the clipboard to show its view • Save and restore clipboard content – click on the save button to write the clipboard content to disk – clipboard is automatically restored when HeuristicLab is started the next time HeuristicLab Tutorial http://dev.heuristiclab.com 27

  3. Clipboard drag & drop here to add algorithms, problems, batch runs, experiments, etc. HeuristicLab Tutorial http://dev.heuristiclab.com 28

  4. Start, Pause, Resume, Stop, Reset HeuristicLab Tutorial http://dev.heuristiclab.com 29

  5. Compare Runs HeuristicLab Tutorial http://dev.heuristiclab.com 30

  6. Analyze Runs • HeuristicLab provides interactive views to analyze and compare all runs of a run collection – textual analysis • RunCollection Tabular View – graphical analysis • RunCollection BubbleChart • RunCollection BoxPlots • Filtering is automatically applied to all open run collection views HeuristicLab Tutorial http://dev.heuristiclab.com 31

  7. Runs – Tabular View HeuristicLab Tutorial http://dev.heuristiclab.com 32

  8. Runs – Tabular View • Sort columns – click on column header to sort column – Ctrl-click on column header to sort multiple columns • Show or hide columns – right-click on table to open dialog to show or hide columns • Compute statistical values – select multiple numerical values to see count, sum, minimum, maximum, average and standard deviation • Select, copy and paste into other applications HeuristicLab Tutorial http://dev.heuristiclab.com 33

  9. Runs – BubbleChart HeuristicLab Tutorial http://dev.heuristiclab.com 34

  10. Runs – BubbleChart • Choose values to plot – choose which values to show on the x-axis, the y-axis and as bubble size – possible values are all parameter settings and results • Add jitter – add jitter to separate overlapping bubbles • Zoom in and out – click on Zoom and click and drag in the chart area to zoom in – double click on the chart area background or on the circle buttons beside the scroll bars to zoom out • Color bubbles – click on Select, choose a color and click and drag in the chart area to select and color bubbles – apply coloring automatically by clicking on the axis coloring buttons • Show runs – double click on a bubble to open its run • Export image – right-click to open context menu to copy or save image – save image as pixel (BMP, JPG, PNG, GIF, TIF) or vector graphics (EMF) • Show box plots – right-click to open context menu to show box plots view HeuristicLab Tutorial http://dev.heuristiclab.com 35

  11. Runs – BoxPlots HeuristicLab Tutorial http://dev.heuristiclab.com 36

  12. Runs – BoxPlots • Choose values to plot – choose which values to show on the x-axis and y-axis – possible values are all parameter settings and results • Zoom in and out – click on Zoom and click and drag in the chart area to zoom in – double click on the chart area background or on the circle buttons beside the scroll bars to zoom out • Show or hide statistical values – click on the lower left button to show or hide statistical values • Export image – right-click to open context menu to copy or save image – save image as pixel (BMP, JPG, PNG, GIF, TIF) or vector graphics (EMF) HeuristicLab Tutorial http://dev.heuristiclab.com 37

  13. Runs – Multi-Line Chart HeuristicLab Tutorial http://dev.heuristiclab.com 38

  14. Filter Runs HeuristicLab Tutorial http://dev.heuristiclab.com 39

  15. Multi-core CPUs and Parallelization • Parallel execution of optimizers in experiments – optimizers in an experiment are executed sequentially from top to bottom per default – experiments support parallel execution of their optimizers – select a not yet executed optimizer and start it manually to utilize another core – execution of one of the next optimizers is started automatically after an optimizer is finished • Parallel execution of algorithms – HeuristicLab provides special operators for parallelization – engines decide how to execute parallel operations – sequential engine executes everything sequentially – parallel engine executes parallel operations on multiple cores – Hive engine (under development) executes parallel operations on multiple computers – all implemented algorithms support parallel solution evaluation HeuristicLab Tutorial http://dev.heuristiclab.com 40

  16. Parallel Execution of Experiments 2. start other optimizers 1. start experiment HeuristicLab Tutorial http://dev.heuristiclab.com 41

  17. Parallel Execution of Algorithms HeuristicLab Tutorial http://dev.heuristiclab.com 42

  18. Analyzers • Special operators for analysis purposes – are executed after each iteration – serve as general purpose extension points of algorithms – can be selected and parameterized in the algorithm – perform algorithm-specific and/or problem-specific tasks – some analyzers are quite costly regarding runtime and memory – implementing and adding custom analyzers is easy • Examples – TSPAlleleFrequencyAnalyzer – TSPPopulationDiversityAnalyzer – SuccessfulOffspringAnalyzer – SymbolicDataAnalysisVariableFrequencyAnalyzer – SymbolicRegressionSingleObjectiveTrainingBestSolutionAnalyzer – … HeuristicLab Tutorial http://dev.heuristiclab.com 43

  19. Analyzers HeuristicLab Tutorial http://dev.heuristiclab.com 44

  20. TSPAlleleFrequencyAnalyzer HeuristicLab Tutorial http://dev.heuristiclab.com 45

  21. TSPPopulationDiversityAnalyzer HeuristicLab Tutorial http://dev.heuristiclab.com 46

  22. Building User-Defined Algorithms • Operator graphs – algorithms are represented as operator graphs – operator graphs of user-defined algorithms can be changed – algorithms can be defined in the graphical algorithm designer – use the menu to convert a standard algorithm into a user-defined algorithm • Operators sidebar – drag & drop operators into an operator graph • Programmable operators – add programmable operators in order to implement custom logic in an algorithm – no additional development environment needed • Debug algorithms – use the debug engine to obtain detailed information during algorithm execution HeuristicLab Tutorial http://dev.heuristiclab.com 47

  23. Building User-Defined Algorithms HeuristicLab Tutorial http://dev.heuristiclab.com 48

  24. Building User-Defined Algorithms HeuristicLab Tutorial http://dev.heuristiclab.com 49

  25. Programmable Operators HeuristicLab Tutorial http://dev.heuristiclab.com 50

  26. Scripts HeuristicLab Tutorial http://dev.heuristiclab.com 51

  27. Debugging Algorithms HeuristicLab Tutorial http://dev.heuristiclab.com 52

  28. Agenda • Objectives of the Tutorial • Introduction • Where to get HeuristicLab? • Plugin Infrastructure • Graphical User Interface • Available Algorithms & Problems • Demonstration Part I: Working with HeuristicLab • Demonstration Part II: Data-based Modeling • Some Additional Features • Planned Features • Team • Suggested Readings • Bibliography • Questions & Answers HeuristicLab Tutorial http://dev.heuristiclab.com 53

  29. Demonstration Part II: Data-based Modeling • Introduction • Regression with HeuristicLab • Model simplification and export • Variable relevance analysis • Classification with HeuristicLab HeuristicLab Tutorial http://dev.heuristiclab.com 54

  30. Introduction to Data-based Modeling • Dataset: Matrix (x i,j ) i=1..N, j= 1..K – N observations of K input variables – x i,j = i-th observation of j-th variable – Additionally: Vector of labels (y 1 … y N ) T • Goal: learn association of input variable values to labels HeuristicLab Tutorial http://dev.heuristiclab.com 55

  31. Data Analysis in HeuristicLab • Symbolic regression and classification using genetic programming • External Libraries: – Linear Regression, Logistic Regression, – k-Nearest Neighbours, k-Means, – Random Forest, Support Vector Machines, Neural Networks, Gaussian Processes HeuristicLab Tutorial http://dev.heuristiclab.com 56

  32. Case Study: Regression • Poly-10 benchmark problem dataset – 10 input variables x 1 … x 10 – y = x 1 x 2 + x 3 x 4 + x 5 x 6 + x 1 x 7 x 9 + x 3 x 6 x 10 – non-linear modeling approach necessary – frequently used in GP literature – available as benchmark problem instance in HeuristicLab HeuristicLab Tutorial http://dev.heuristiclab.com 57

  33. Demonstration • problem configuration – data import – target and input variables – data partitions (training and test) • algorithm configuration • analysis of results – accuracy metrics – visualization of model output HeuristicLab Tutorial http://dev.heuristiclab.com 58

  34. Linear Regression • Create new algorithm HeuristicLab Tutorial http://dev.heuristiclab.com 59

  35. Import Data from CSV-File HeuristicLab Tutorial http://dev.heuristiclab.com 60

  36. Inspect and Configure Dataset HeuristicLab Tutorial http://dev.heuristiclab.com 61

  37. Inspect Imported Data HeuristicLab Tutorial http://dev.heuristiclab.com 62

  38. Set Target Variable HeuristicLab Tutorial http://dev.heuristiclab.com 63

  39. Select Input Variables HeuristicLab Tutorial http://dev.heuristiclab.com 64

  40. Configure Training and Test Partitions HeuristicLab Tutorial http://dev.heuristiclab.com 65

  41. Run Linear Regression HeuristicLab Tutorial http://dev.heuristiclab.com 66

  42. Inspect Results HeuristicLab Tutorial http://dev.heuristiclab.com 67

  43. Inspect Scatterplot of Predicted and Target Values HeuristicLab Tutorial http://dev.heuristiclab.com 68

  44. Inspect Linechart HeuristicLab Tutorial http://dev.heuristiclab.com 69

  45. Inspect the Model HeuristicLab Tutorial http://dev.heuristiclab.com 70

  46. Symbolic Regression with HeuristicLab • Linear regression produced an inaccurate model. • Next: produce a nonlinear symbolic regression model using genetic programming • Genetic programming – evolve variable-length models – model representation: symbolic expression tree – structure and model parameters are evolved side-by-side – white-box models - x3 + x1 * x1 x2 HeuristicLab Tutorial http://dev.heuristiclab.com 71

  47. Symbolic Regression with HeuristicLab • Demonstration – problem configuration – function set and terminal set – model size constraints – evaluation • Algorithm configuration – selection – mutation • Analysis of results – model accuracy – model structure and parameters HeuristicLab Tutorial http://dev.heuristiclab.com 72

  48. Create New Genetic Algorithm HeuristicLab Tutorial http://dev.heuristiclab.com 73

  49. Create New Symbolic Regression Problem HeuristicLab Tutorial http://dev.heuristiclab.com 74

  50. Import Data HeuristicLab Tutorial http://dev.heuristiclab.com 75

  51. Inspect Data and Configure Dataset HeuristicLab Tutorial http://dev.heuristiclab.com 76

  52. Set Target and Input Variables HeuristicLab Tutorial http://dev.heuristiclab.com 77

  53. Configure Maximal Model Depth and Length HeuristicLab Tutorial http://dev.heuristiclab.com 78

  54. Configure Function Set (Grammar) HeuristicLab Tutorial http://dev.heuristiclab.com 79

  55. Configure Function Set (Grammar) HeuristicLab Tutorial http://dev.heuristiclab.com 80

  56. Configure Algorithm Parameters HeuristicLab Tutorial http://dev.heuristiclab.com 81

  57. Configure Mutation Operator HeuristicLab Tutorial http://dev.heuristiclab.com 82

  58. Configure Selection Operator HeuristicLab Tutorial http://dev.heuristiclab.com 83

  59. Configure Tournament Group Size HeuristicLab Tutorial http://dev.heuristiclab.com 84

  60. Start Algorithm and Inspect Results HeuristicLab Tutorial http://dev.heuristiclab.com 85

  61. Inspect Quality Chart HeuristicLab Tutorial http://dev.heuristiclab.com 86

  62. Inspect Best Model on Training Partition HeuristicLab Tutorial http://dev.heuristiclab.com 87

  63. Inspect Linechart of Best Model on Training Partition HeuristicLab Tutorial http://dev.heuristiclab.com 88

  64. Inspect Structure of Best Model on Training Partition HeuristicLab Tutorial http://dev.heuristiclab.com 89

  65. Model Simplification and Export • Demonstration – automatic simplification – visualization of node impacts – manual simplification • online update of results – model export • Excel • MATLAB • LaTeX HeuristicLab Tutorial http://dev.heuristiclab.com 90

  66. Detailed Model Analysis and Simplification HeuristicLab Tutorial http://dev.heuristiclab.com 91

  67. Symbolic Simplification and Node Impacts HeuristicLab Tutorial http://dev.heuristiclab.com 92

  68. Manual Simplification double-click nodes HeuristicLab Tutorial http://dev.heuristiclab.com 93

  69. Automatic Symbolic Simplification HeuristicLab Tutorial http://dev.heuristiclab.com 94

  70. Textual Representations Are Also Available • Use ViewHost to switch to textual representation view. HeuristicLab Tutorial http://dev.heuristiclab.com 95

  71. Default Textual Representation for Model Export HeuristicLab Tutorial http://dev.heuristiclab.com 96

  72. Textual Representation for Export to LaTeX HeuristicLab Tutorial http://dev.heuristiclab.com 97

  73. LaTeX Export HeuristicLab Tutorial http://dev.heuristiclab.com 98

  74. Variable Relevance Analysis • Which variables are important for correct predictions? • Demonstration – Variable frequency analyzer – symbol frequency analyzer – variable impacts HeuristicLab Tutorial http://dev.heuristiclab.com 99

  75. Inspect Variable Frequency Chart HeuristicLab Tutorial http://dev.heuristiclab.com 100

More recommend