I t Introduction to d ti t Evolutionary Algorithms Federico - PDF document

4/17/2020 I t Introduction to d ti t Evolutionary Algorithms Federico Nesti, f.nesti@santannapisa.it Layout • Evolutionary Algorithms basics • Evolutionary Algorithms for Control • CMA-ES 1

4/17/2020 Layout • Evolutionary Algorithms basics • Evolutionary Algorithms for Control • CMA-ES Evolutionary Algorithms Evolutionary algorithms are Black-box, gradient-free optimization methods. They are biologically inspired, and rely on the concept of evolution and survival of the fittest. 2

4/17/2020 Evolutionary Algorithms Evolutionary Algorithms 3

4/17/2020 Evolutionary Algorithms 1 2 3 Evolutionary Algorithms 1 2 3 4

4/17/2020 Evolutionary Algorithms Step 4 – Reproduction/ recom bination : the survived individuals reproduce and their genes are recombined in the offspring to refill the population. Parents Offsprings Evolutionary Algorithms Step 5 – Mutation : stochastically, it could happen that a mutation occurs in one or more of the individuals. 5

4/17/2020 Evolutionary Algorithms I nitialization I terate steps 2 -5 : I terate steps 2 5 : a Generation is Ranking and composed of survival evaluation, survival, Reproduction reproduction and mutation. mutation Mutation Applications of EAs Com puter-Aided Design p g https://www.youtube.com/watch?v=aR5N2Jl8k14&t=207s 6

4/17/2020 Applications of EAs Evolvable Electronics Evolvable Electronics Applications of EAs Molecular Design Molecular Design 7

4/17/2020 Applications of EAs …and m any m ore! …and m any m ore! • Image Processing • Climatology • Finance and Economics • Social Sciences • Quality Control • Biological Applications Evolutionary Algorithms for Control 8

4/17/2020 EAs for Control Run for M generations Observations, Survival, reward Actions Mutation, Recombination Ranking Fitness Reward Evolutionary Algorithms There are many evolutionary algorithms, and are classified in lots of different categories. Some of them are intuitively described in this great blog post. In this lecture we are going to use only CMA-ES (Covariance Matrix Adaptation – Evolutionary Strategy), one of the most popular Evolutionary Algorithms For full one of the most popular Evolutionary Algorithms. For full implementation details, refer to this tutorial. 9

4/17/2020 Covariance Matrix Adaptation Iterate for M generations Covariance Matrix Adaptation Iterate for M generations 10

4/17/2020 Covariance Matrix Adaptation Iterate for M generations Covariance Matrix Adaptation Iterate for M generations 11

4/17/2020 Covariance Matrix Adaptation Iterate for M generations Reinforcement Learning Car Acceleration, Acceleration, Sh ll Shallow NN NN Steer 12

4/17/2020 CMA-ES for Car Control No obstacles 60 generations 20 individuals Keep 10 best CMA-ES for Car Control With obstacles Same hyperp. 13

4/17/2020 Conclusions Evolutionary Algorithms for RL are promising, since • The search for the optimal solution is not done sequentially, but « in parallel» for each different solution . This allows a much more broader exploration and could lead to better solutions. • There is no need to have a strong mathematical background to optimize a network! background to optimize a network! 14