understanding the impact of entropy on policy optimization
play

Understanding the impact of entropy on policy optimization Zafarali - PowerPoint PPT Presentation

Feb 05, 2023 •128 likes •205 views

Understanding the impact of entropy on policy optimization Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, Dale Schuurmans bit.ly/2HQvGoQ zafarali.ahmed@mail.mcgill.ca Why should we understand policy optimization? What is policy

  1. Understanding the impact of entropy on policy optimization Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, Dale Schuurmans bit.ly/2HQvGoQ zafarali.ahmed@mail.mcgill.ca

  2. Why should we understand policy optimization? What is policy optimization? Find parameterized policy that maximizes rewards. (1) Collect data + calculate objective (2) Take gradient + update policy parameters Why is it difficult? Bad gradient estimates? Difficult geometry? Poor conditioning? Not enough “Exploration”?

  3. Contribution 1: How do we study high dim objective functions? STEP 2: How does objective change along random perturbations? STEP 1: Collect random perturbations of (+, +) objective (+, -) (-, -) θ 0

  4. Contribution 1: How do we study high dim objective functions? Examples @ A Local Optimum @ A Saddle Point

  5. Contribution 2: Why does entropy regularization help? Experiments on exact grid worlds and Mujoco 4.5 Conclusion: Even the absence of gradient estimation error, policy entropy helps by smoothing the objective function:

  6. Understanding the impact of entropy on policy optimization Read the paper! bit.ly/2HQvGoQ Come see poster! Poster # 29 TODAY - 6.30 PM Pacific Ballroom Chat with me! zafarali.ahmed@mail.mcgill.ca

Recommend

Entropy, Relative Entropy, Cross Entropy Entropy Entropy, H(x) is a measure of the uncertainty of

Entropy, Relative Entropy, Cross Entropy Entropy Entropy, H(x) is a measure of the uncertainty of

Entropy, Relative Entropy, Cross Entropy Entropy Entropy, H(x) is a measure of the uncertainty of a discrete random variable. Properties: H(x) >= 0 Entropy Entropy Lesser the probability for an event, larger the entropy.

471 views • 21 slides
Entropy and The Second Law of Thermodynamics Entropy (S)

Entropy and The Second Law of Thermodynamics Entropy (S)

Entropy and The Second Law of Thermodynamics Entropy (S) Entropy is o8en related to disorder but not strictly correct Entropy is a measure

351 views • 8 slides
Formal Modeling in Cognitive Science Lecture 25: Entropy, Joint Entropy, Conditional Entropy 1

Formal Modeling in Cognitive Science Lecture 25: Entropy, Joint Entropy, Conditional Entropy 1

Entropy Entropy Formal Modeling in Cognitive Science Lecture 25: Entropy, Joint Entropy, Conditional Entropy 1 Entropy Entropy and Information Joint Entropy Frank Keller Conditional Entropy School of Informatics University of Edinburgh

81 views • 4 slides
ABAWD Policy Training UNDERSTANDING THE NEW SNAP TIME LIMIT AND ITS IMPACT IN MARYLAND MARYLAND

ABAWD Policy Training UNDERSTANDING THE NEW SNAP TIME LIMIT AND ITS IMPACT IN MARYLAND MARYLAND

ABAWD Policy Training UNDERSTANDING THE NEW SNAP TIME LIMIT AND ITS IMPACT IN MARYLAND MARYLAND HUNGER SOLUTIONS AND MANNA FOOD CENTER, FEB 4, 2016 Training Overview National Overview of ABAWD Policy Definitions, Criteria and Eligible

432 views • 9 slides
WELCOME The Politics of Food: Understanding the Relationship between Food Policy Issues,

WELCOME The Politics of Food: Understanding the Relationship between Food Policy Issues,

WELCOME The Politics of Food: Understanding the Relationship between Food Policy Issues, Societal Impact Factors and Consumer Food Behaviors The Politics of Food: Understanding the Relationship between Food Policy Issues, Societal Impact

433 views • 31 slides
Entropy Coding Definition of Entropy Three Entropy coding techniques: (taken from the

Entropy Coding Definition of Entropy Three Entropy coding techniques: (taken from the

Outline Entropy Coding Definition of Entropy Three Entropy coding techniques: (taken from the Technion) Huffman coding Arithmetic coding Lempel-Ziv coding 2 Entropy Definitions Alphabet : A finite set containing at least

402 views • 10 slides
RTI PLC Understanding our Impact Spring 2016 RTI PLC Agenda What factors impact student

RTI PLC Understanding our Impact Spring 2016 RTI PLC Agenda What factors impact student

RTI PLC Understanding our Impact Spring 2016 RTI PLC Agenda What factors impact student achievement and how do these relate to RTI? How do RTI and the areas of impact strengthen: Writing Math instruction and interventions

387 views • 11 slides
Koszul/Souriau Fisher Metric Spaces & Optimization by Maximum Entropy: Hessian Information

Koszul/Souriau Fisher Metric Spaces & Optimization by Maximum Entropy: Hessian Information

Koszul/Souriau Fisher Metric Spaces & Optimization by Maximum Entropy: Hessian Information Geometry, Lie Group Thermodynamics & Poincar'Marle'Souriau Equation Frdric

1.18k views • 83 slides
Exponential cone in MOSEK ISMP2018, Relative Entropy Optimization, 6 July 2018 Micha l

Exponential cone in MOSEK ISMP2018, Relative Entropy Optimization, 6 July 2018 Micha l

Exponential cone in MOSEK ISMP2018, Relative Entropy Optimization, 6 July 2018 Micha l Adamaszek, MOSEK ApS www.mosek.com MOSEK linear conic solver: SOCP, SDP, EXP, POW, primal/dual simplex for LPs, convex QPs, +

824 views • 22 slides
Road detection via entropy By Anna Zaidman 1 1 What is entropy? Entropy is a mathematically

Road detection via entropy By Anna Zaidman 1 1 What is entropy? Entropy is a mathematically

Road detection via entropy By Anna Zaidman 1 1 What is entropy? Entropy is a mathematically - defined thermodynamic quantity that helps to account for the flow of energy through a thermodynamic process. 2 What is

467 views • 13 slides
Predictor-Corrector Policy Optimization (PicCoLO) Ching-An Cheng #$ , Xinyan Yan # , Nathan

Predictor-Corrector Policy Optimization (PicCoLO) Ching-An Cheng #$ , Xinyan Yan # , Nathan

Predictor-Corrector Policy Optimization (PicCoLO) Ching-An Cheng #$ , Xinyan Yan # , Nathan Ratliff $ , Byron Boots #$ Jun 11 2019 @ ICML # $ Policy optimization We consider episodic learning (MDP) Optimize a policy

290 views • 27 slides
Relative entropy optimization in quantum information Omar Fawzi ICMP 2018, Montr eal 1/11

Relative entropy optimization in quantum information Omar Fawzi ICMP 2018, Montr eal 1/11

Relative entropy optimization in quantum information Omar Fawzi ICMP 2018, Montr eal 1/11 Quantum relative entropie s For classical states (i.e., prob. distributions) P and Q on X P ( x ) log P ( x ) D ( P Q ) := Q ( x ) x X 2/11

615 views • 22 slides
Understanding Optimization Phase Interactions to Reduce the Phase Order Search Space Michael

Understanding Optimization Phase Interactions to Reduce the Phase Order Search Space Michael

Understanding Optimization Phase Interactions to Reduce the Phase Order Search Space Michael Jantz Prasad Kulkarni (Advisor) 1 Introduction 2 Optimization Phases Conventional optimizing compilers contain several optimization phases

848 views • 67 slides
Outline Continuous Optimization DM812 METAHEURISTICS Lecture 12 1. Model Based Metaheuristics

Outline Continuous Optimization DM812 METAHEURISTICS Lecture 12 1. Model Based Metaheuristics

Model Based Metaheuristics Outline Continuous Optimization DM812 METAHEURISTICS Lecture 12 1. Model Based Metaheuristics Cross Entropy Method Cross Entropy Method Continuous Optimization 2. Continuous Optimization Marco Chiarandini

229 views • 5 slides
Prevalence and Impact of Low-Entropy Packing Schemes in the Malware Ecosystem Alessandro

Prevalence and Impact of Low-Entropy Packing Schemes in the Malware Ecosystem Alessandro

Prevalence and Impact of Low-Entropy Packing Schemes in the Malware Ecosystem Alessandro Mantovani (EURECOM), Simone Aonzo (UniGe), Xabier Ugarte Pedrero (CISCO), Alessio Merlo (UniGe), Davide Balzarotti (EURECOM) 1 Packing 2 Scope / Packing

686 views • 24 slides
Prevalence and Impact of Low-Entropy Packing Schemes in the Malware Ecosystem Alessandro

Prevalence and Impact of Low-Entropy Packing Schemes in the Malware Ecosystem Alessandro

Prevalence and Impact of Low-Entropy Packing Schemes in the Malware Ecosystem Alessandro Mantovani (EURECOM), Simone Aonzo (UniGe), Xabier Ugarte Pedrero (CISCO), Alessio Merlo (UniGe), Davide Balzarotti (EURECOM) 1 Packing 2 Scope / Packing

256 views • 23 slides
Dr. Jennifer Zwicker HOW DOES DATA IMPACT POLICY? Policy/clinical impact Evaluation

Dr. Jennifer Zwicker HOW DOES DATA IMPACT POLICY? Policy/clinical impact Evaluation

How does data impact policy? Dr. Jennifer Zwicker HOW DOES DATA IMPACT POLICY? Policy/clinical impact Evaluation Intervention research and clinical application Needs assessment DATA AND NEEDS OF PERSONS WITH DISABILITIES NEEDS ASSESSMENT

596 views • 28 slides
Optimization-Based Control: Direct Collocation Methods for Trajectory and Policy Optimization CS

Optimization-Based Control: Direct Collocation Methods for Trajectory and Policy Optimization CS

Optimization-Based Control: Direct Collocation Methods for Trajectory and Policy Optimization CS 287: Advanced Robotics, Fall 2019 Guest Lecture Igor Mordatch Overview Previously: Locally optimal control (shooting vs. collocation)

1.54k views • 138 slides
E N T R O P Y S T R U C T U R E Entropy versus resolution Tomasz Downarowicz MOTIVATION

E N T R O P Y S T R U C T U R E Entropy versus resolution Tomasz Downarowicz MOTIVATION

E N T R O P Y S T R U C T U R E Entropy versus resolution Tomasz Downarowicz MOTIVATION Entropy measures exponential complexity in a topological dynamical system: topological entropy very crude, entropy function on invariant

556 views • 23 slides
Chapter 2 Entropy, Relative Entropy, and Mutual Infor- mation Peng-Hua Wang Graduate Institute

Chapter 2 Entropy, Relative Entropy, and Mutual Infor- mation Peng-Hua Wang Graduate Institute

Chapter 2 Entropy, Relative Entropy, and Mutual Infor- mation Peng-Hua Wang Graduate Institute of Communication Engineering National Taipei University Chapter Outline Chap. 2 Entropy, Relative Entropy, and Mutual Information 2.1 Entropy 2.2

753 views • 51 slides
Huffman Encoding 13-Oct-11 Entropy Entropy is a measure of information content: the number of

Huffman Encoding 13-Oct-11 Entropy Entropy is a measure of information content: the number of

Huffman Encoding 13-Oct-11 Entropy Entropy is a measure of information content: the number of bits actually required to store data. Entropy is sometimes called a measure of surprise A highly predictable sequence contains little actual

293 views • 16 slides
ARE THE SHANNON ENTROPY AND RESIDUAL ENTROPY SYNONYMS? H = R 0 ? MARKO POPOVIC DEPARTMENT OF

ARE THE SHANNON ENTROPY AND RESIDUAL ENTROPY SYNONYMS? H = R 0 ? MARKO POPOVIC DEPARTMENT OF

ARE THE SHANNON ENTROPY AND RESIDUAL ENTROPY SYNONYMS? H = R 0 ? MARKO POPOVIC DEPARTMENT OF CHEMISTRY AND BIOCHEMISTRY, BYU, PROVO, UT 84602, POPOVIC.PASA@GMAIL.COM Thermodynamic entropy - S (J/K) At T=0K ideal crystal imperfect

560 views • 21 slides
Infotheory for Statistics and Learning Lecture 1 Entropy Relative entropy Mutual

Infotheory for Statistics and Learning Lecture 1 Entropy Relative entropy Mutual

Infotheory for Statistics and Learning Lecture 1 Entropy Relative entropy Mutual information f -divergence Mikael Skoglund 1/16 Entropy Over ( R , B ) , consider a discrete RV X with all probability in a countable set X B ,

538 views • 8 slides
Collective Impact for Youth Understanding how the principles of collective impact can support

Collective Impact for Youth Understanding how the principles of collective impact can support

Collective Impact for Youth Understanding how the principles of collective impact can support OYDC work and beyond 1 About Education Northwest Education Northwest is a regionally based nonprofit that works throughout the nation to create strong

1.15k views • 57 slides

More recommend