approximate q learning update
play

Approximate Q-Learning Update Initialize weight for each feature to - PowerPoint PPT Presentation

Dec 23, 2022 •130 likes •249 views

<latexit

  1. <latexit sha1_base64="cCR4OZ9l/OETB0Xei2CIfbgOzM=">ACDHicbVDLSsNAFJ3UV62vqEs3g1WoICURQV0Uim5ctmBsoY1hMp20QyeTMDNRSsgPuPFX3LhQcesHuPNvnLZaOuBC4dz7uXe/yYUaks69soLCwuLa8UV0tr6xubW+b2zq2MEoGJgyMWibaPJGUE0dRxUg7FgSFPiMtf3g19lv3REga8Rs1iokboj6nAcVIackzD5oVeYyOYA12ZRJ6Ka3Z2V3KMxh4dOo8eNQzy1bVmgDOEzsnZCj4Zlf3V6Ek5BwhRmSsmNbsXJTJBTFjGSlbiJjPAQ9UlHU45CIt108k0GD7XSg0EkdHEFJ+rviRSFUo5CX3eGSA3krDcW/M6iQrO3ZTyOFGE4+miIGFQRXAcDexRQbBiI0QFlTfCvEACYSVDrCkQ7BnX54nzkn1omo3T8v1yzyNItgD+6ACbHAG6uAaNIADMHgEz+AVvBlPxovxbnxMWwtGPrML/sD4/AG0/pmn</latexit> <latexit sha1_base64="cCR4OZ9l/OETB0Xei2CIfbgOzM=">ACDHicbVDLSsNAFJ3UV62vqEs3g1WoICURQV0Uim5ctmBsoY1hMp20QyeTMDNRSsgPuPFX3LhQcesHuPNvnLZaOuBC4dz7uXe/yYUaks69soLCwuLa8UV0tr6xubW+b2zq2MEoGJgyMWibaPJGUE0dRxUg7FgSFPiMtf3g19lv3REga8Rs1iokboj6nAcVIackzD5oVeYyOYA12ZRJ6Ka3Z2V3KMxh4dOo8eNQzy1bVmgDOEzsnZCj4Zlf3V6Ek5BwhRmSsmNbsXJTJBTFjGSlbiJjPAQ9UlHU45CIt108k0GD7XSg0EkdHEFJ+rviRSFUo5CX3eGSA3krDcW/M6iQrO3ZTyOFGE4+miIGFQRXAcDexRQbBiI0QFlTfCvEACYSVDrCkQ7BnX54nzkn1omo3T8v1yzyNItgD+6ACbHAG6uAaNIADMHgEz+AVvBlPxovxbnxMWwtGPrML/sD4/AG0/pmn</latexit> <latexit sha1_base64="cCR4OZ9l/OETB0Xei2CIfbgOzM=">ACDHicbVDLSsNAFJ3UV62vqEs3g1WoICURQV0Uim5ctmBsoY1hMp20QyeTMDNRSsgPuPFX3LhQcesHuPNvnLZaOuBC4dz7uXe/yYUaks69soLCwuLa8UV0tr6xubW+b2zq2MEoGJgyMWibaPJGUE0dRxUg7FgSFPiMtf3g19lv3REga8Rs1iokboj6nAcVIackzD5oVeYyOYA12ZRJ6Ka3Z2V3KMxh4dOo8eNQzy1bVmgDOEzsnZCj4Zlf3V6Ek5BwhRmSsmNbsXJTJBTFjGSlbiJjPAQ9UlHU45CIt108k0GD7XSg0EkdHEFJ+rviRSFUo5CX3eGSA3krDcW/M6iQrO3ZTyOFGE4+miIGFQRXAcDexRQbBiI0QFlTfCvEACYSVDrCkQ7BnX54nzkn1omo3T8v1yzyNItgD+6ACbHAG6uAaNIADMHgEz+AVvBlPxovxbnxMWwtGPrML/sD4/AG0/pmn</latexit> <latexit sha1_base64="cCR4OZ9l/OETB0Xei2CIfbgOzM=">ACDHicbVDLSsNAFJ3UV62vqEs3g1WoICURQV0Uim5ctmBsoY1hMp20QyeTMDNRSsgPuPFX3LhQcesHuPNvnLZaOuBC4dz7uXe/yYUaks69soLCwuLa8UV0tr6xubW+b2zq2MEoGJgyMWibaPJGUE0dRxUg7FgSFPiMtf3g19lv3REga8Rs1iokboj6nAcVIackzD5oVeYyOYA12ZRJ6Ka3Z2V3KMxh4dOo8eNQzy1bVmgDOEzsnZCj4Zlf3V6Ek5BwhRmSsmNbsXJTJBTFjGSlbiJjPAQ9UlHU45CIt108k0GD7XSg0EkdHEFJ+rviRSFUo5CX3eGSA3krDcW/M6iQrO3ZTyOFGE4+miIGFQRXAcDexRQbBiI0QFlTfCvEACYSVDrCkQ7BnX54nzkn1omo3T8v1yzyNItgD+6ACbHAG6uAaNIADMHgEz+AVvBlPxovxbnxMWwtGPrML/sD4/AG0/pmn</latexit> Approximate Q-Learning Update Initialize weight for each feature to 0. Every time we take an action, perform this update: The Q-value estimate for (s,a) is the weighted sum of its features: n X Q ( s, a ) = f i ( s, a ) w i i =1

  2. <latexit sha1_base64="DA+nU6WuIiNWRNEhvmCSzynrSI=">ACI3icbZDNSsNAFIUn9a/Wv6pLN4NFqCAlEUFCkU3LluwtDUMJlO2qGTSZiZtJSQh3Hjq7hxocWNC9/FSdqFtl4YOJzvXu7c4aMSmWaX0ZuZXVtfSO/Wdja3tndK+4fPMogEpg0cAC0XaRJIxy0lRUMdIOBUG+y0jLHd6lvDUiQtKAP6hJSLo+6nPqUYyUtpziTaMsz9AprEJbRr4T06qVPMU8gZ5DZ2Ts0JSOCI7HCbRxL1DQy5BTLJkVMyu4LKy5KIF51Z3i1O4FOPIJV5ghKTuWGapujISimJGkYEeShAgPUZ90tOTIJ7IbZ0cm8EQ7PegFQj+uYOb+noiRL+XEd3Wnj9RALrLU/I91IuVdWPKw0gRjmeLvIhBFcA0MdijgmDFJlogLKj+K8QDJBWOteCDsFaPHlZNM8r1xWrcVGq3c7TyIMjcAzKwAKXoAbuQR0AQbP4BW8gw/jxXgzpsbnrDVnzGcOwZ8yvn8AzemiEA=</latexit> <latexit sha1_base64="DA+nU6WuIiNWRNEhvmCSzynrSI=">ACI3icbZDNSsNAFIUn9a/Wv6pLN4NFqCAlEUFCkU3LluwtDUMJlO2qGTSZiZtJSQh3Hjq7hxocWNC9/FSdqFtl4YOJzvXu7c4aMSmWaX0ZuZXVtfSO/Wdja3tndK+4fPMogEpg0cAC0XaRJIxy0lRUMdIOBUG+y0jLHd6lvDUiQtKAP6hJSLo+6nPqUYyUtpziTaMsz9AprEJbRr4T06qVPMU8gZ5DZ2Ts0JSOCI7HCbRxL1DQy5BTLJkVMyu4LKy5KIF51Z3i1O4FOPIJV5ghKTuWGapujISimJGkYEeShAgPUZ90tOTIJ7IbZ0cm8EQ7PegFQj+uYOb+noiRL+XEd3Wnj9RALrLU/I91IuVdWPKw0gRjmeLvIhBFcA0MdijgmDFJlogLKj+K8QDJBWOteCDsFaPHlZNM8r1xWrcVGq3c7TyIMjcAzKwAKXoAbuQR0AQbP4BW8gw/jxXgzpsbnrDVnzGcOwZ8yvn8AzemiEA=</latexit> <latexit sha1_base64="DA+nU6WuIiNWRNEhvmCSzynrSI=">ACI3icbZDNSsNAFIUn9a/Wv6pLN4NFqCAlEUFCkU3LluwtDUMJlO2qGTSZiZtJSQh3Hjq7hxocWNC9/FSdqFtl4YOJzvXu7c4aMSmWaX0ZuZXVtfSO/Wdja3tndK+4fPMogEpg0cAC0XaRJIxy0lRUMdIOBUG+y0jLHd6lvDUiQtKAP6hJSLo+6nPqUYyUtpziTaMsz9AprEJbRr4T06qVPMU8gZ5DZ2Ts0JSOCI7HCbRxL1DQy5BTLJkVMyu4LKy5KIF51Z3i1O4FOPIJV5ghKTuWGapujISimJGkYEeShAgPUZ90tOTIJ7IbZ0cm8EQ7PegFQj+uYOb+noiRL+XEd3Wnj9RALrLU/I91IuVdWPKw0gRjmeLvIhBFcA0MdijgmDFJlogLKj+K8QDJBWOteCDsFaPHlZNM8r1xWrcVGq3c7TyIMjcAzKwAKXoAbuQR0AQbP4BW8gw/jxXgzpsbnrDVnzGcOwZ8yvn8AzemiEA=</latexit> <latexit sha1_base64="DA+nU6WuIiNWRNEhvmCSzynrSI=">ACI3icbZDNSsNAFIUn9a/Wv6pLN4NFqCAlEUFCkU3LluwtDUMJlO2qGTSZiZtJSQh3Hjq7hxocWNC9/FSdqFtl4YOJzvXu7c4aMSmWaX0ZuZXVtfSO/Wdja3tndK+4fPMogEpg0cAC0XaRJIxy0lRUMdIOBUG+y0jLHd6lvDUiQtKAP6hJSLo+6nPqUYyUtpziTaMsz9AprEJbRr4T06qVPMU8gZ5DZ2Ts0JSOCI7HCbRxL1DQy5BTLJkVMyu4LKy5KIF51Z3i1O4FOPIJV5ghKTuWGapujISimJGkYEeShAgPUZ90tOTIJ7IbZ0cm8EQ7PegFQj+uYOb+noiRL+XEd3Wnj9RALrLU/I91IuVdWPKw0gRjmeLvIhBFcA0MdijgmDFJlogLKj+K8QDJBWOteCDsFaPHlZNM8r1xWrcVGq3c7TyIMjcAzKwAKXoAbuQR0AQbP4BW8gw/jxXgzpsbnrDVnzGcOwZ8yvn8AzemiEA=</latexit> AQL Update Details • The weighted sum of features is equivalent to a dot product between the feature and weight vectors: n X Q ( s, a ) = f i ( s, a ) w i = ~ w · f ( s, a ) i =1 • The correction term is the same for all features. • The correction to each feature is weighted by how “active” that feature was.

  3. Exercise: Feature Q-Update Reward eating food: discount: .95 +10 learning rate: .3 Reward for losing: -500 • Suppose PacMan is considering the up action. Old Feature Values: w bias = 1 w ghosts = -20 w food = 2 Features: w eats = 4 • closest-food • bias • eats-food • #-of-ghosts-1-step-away

  4. Notes on Approximate Q-Learning • Learns weights for a tiny number of features. • Every feature’s value is update every step. • No longer tracking values for individual (s,a) pairs. • (s,a) value estimates are calculated from features. • The weight update is a form of gradient descent. • We’ve seen this before. • We’re performing a variant of linear regression. • Feature extraction is a type of basis change. • We’ll see these again.

Recommend

Approximate Q-Learning 3/23/18 On-Policy Learning (SARSA) Instead of updating based on the best

Approximate Q-Learning 3/23/18 On-Policy Learning (SARSA) Instead of updating based on the best

Approximate Q-Learning 3/23/18 On-Policy Learning (SARSA) Instead of updating based on the best action from the next state, update based on the action your current policy actually takes from the next state. SARSA update: When would this be

572 views • 19 slides
Sensors Approximate computing approximate edge detection Machine learning hidden

Sensors Approximate computing approximate edge detection Machine learning hidden

Uncertain&lt; T &gt; A First-Order Type for Uncertain Data James Bornholt Australian National University Todd Mytkowicz Microsoft Research Kathryn S. McKinley Microsoft Research Sensors Approximate computing

1.19k views • 97 slides
Approximate Q-Learning 2/24/17 State Value V(s) vs. Action Value Q(s,a) &quot; # X t r t

Approximate Q-Learning 2/24/17 State Value V(s) vs. Action Value Q(s,a) &quot; # X t r t

Approximate Q-Learning 2/24/17 State Value V(s) vs. Action Value Q(s,a) &quot; # X t r t V = E t =0 Either way, value is the sum of future discounted rewards (assuming the agent behaves optimally): V(s) : after being in state s

404 views • 17 slides
Approximate Inference Part 1 of 2 Tom Minka Microsoft Research, Cambridge, UK Machine Learning

Approximate Inference Part 1 of 2 Tom Minka Microsoft Research, Cambridge, UK Machine Learning

Approximate Inference Part 1 of 2 Tom Minka Microsoft Research, Cambridge, UK Machine Learning Summer School 2009 http://mlg.eng.cam.ac.uk/mlss09/ 1 Bayesian paradigm Consistent use of probability theory for representing unknowns

1.05k views • 81 slides
Time Series Forecasting With a Learning Algorithm: An Approximate Dynamic Programming Approach

Time Series Forecasting With a Learning Algorithm: An Approximate Dynamic Programming Approach

Time Series Forecasting With a Learning Algorithm: An Approximate Dynamic Programming Approach Ricardo Collado 1 an Creamer 1 Germ 1 School of Business Stevens Institute of Technology Hoboken, New Jersey R. Collado (Stevens) Learning Time

891 views • 71 slides
Reinforcement Learning: Approximate Dynamic Programming Decision Making Under Uncertainty, Chapter

Reinforcement Learning: Approximate Dynamic Programming Decision Making Under Uncertainty, Chapter

Reinforcement Learning: Approximate Dynamic Programming Decision Making Under Uncertainty, Chapter 10 Christos Dimitrakakis Chalmers November 21, 2013 Christos Dimitrakakis (Chalmers) Reinforcement Learning: Approximate Dynamic Programming

587 views • 22 slides
Probabilistic &amp; Unsupervised Learning Convex Algorithms in Approximate Inference Maneesh

Probabilistic &amp; Unsupervised Learning Convex Algorithms in Approximate Inference Maneesh

Probabilistic &amp; Unsupervised Learning Convex Algorithms in Approximate Inference Maneesh Sahani maneesh@gatsby.ucl.ac.uk Gatsby Computational Neuroscience Unit, and MSc ML/CSML, Dept Computer Science University College London Term 1,

861 views • 41 slides
Backward Analysis via Over-Approximate Abstraction and Under-Approximate Subtraction Alexey

Backward Analysis via Over-Approximate Abstraction and Under-Approximate Subtraction Alexey

Backward Analysis via Over-Approximate Abstraction and Under-Approximate Subtraction Alexey Bakhirkin 1 Josh Berdine 2 Nir Piterman 1 1 University of Leicester, Department of Computer Science 2 Microsoft Research Goal A backwards analysis

733 views • 23 slides
Approximate Kernel Methods and Learning on Aggregates Dino Sejdinovic joint work with Leon Law,

Approximate Kernel Methods and Learning on Aggregates Dino Sejdinovic joint work with Leon Law,

Approximate Kernel Methods and Learning on Aggregates Dino Sejdinovic joint work with Leon Law, Seth Flaxman, Dougal Sutherland, Kenji Fukumizu, Ewan Cameron, Tim Lucas, Katherine Battle (and many others) Department of Statistics University of

580 views • 34 slides
Multiple-Step Greedy Policies in Online and Approximate Reinforcement Learning Neural Information

Multiple-Step Greedy Policies in Online and Approximate Reinforcement Learning Neural Information

... Multiple-Step Greedy Policies in Online and Approximate Reinforcement Learning Neural Information Processing Systems, December 18 Yonathan Efroni 1 Gal Dalal 1 Bruno Scherrer 2 Shie Mannor 1 1 Department of Electrical Engineering,

716 views • 36 slides
Approximate Q-Learning 3-25-16 Exploration policy vs. optimal policy Where do the exploration

Approximate Q-Learning 3-25-16 Exploration policy vs. optimal policy Where do the exploration

Approximate Q-Learning 3-25-16 Exploration policy vs. optimal policy Where do the exploration traces come from? We need some policy for acting in the environment before we understand it. Wed like to get decent rewards while

264 views • 14 slides
Approximate MPC based on machine learning and probabilistic verification Sergio Lucia Technische

Approximate MPC based on machine learning and probabilistic verification Sergio Lucia Technische

Approximate MPC based on machine learning and probabilistic verification Sergio Lucia Technische Universitt Berlin Einstein Center Digital Future www.iot.tu-berlin.de Motivation Solving NMPC problems in real time is still challenging:

1.03k views • 36 slides
Partially Exchangeable Networks and Architectures for Learning Summary Statistics in Approximate

Partially Exchangeable Networks and Architectures for Learning Summary Statistics in Approximate

Partially Exchangeable Networks and Architectures for Learning Summary Statistics in Approximate Bayesian Computation ICML 2019 Samuel Wiqvist Centre for Mathematical Sciences, Lund University, Sweden samuel wiqvist June 13, 2019 Joint

685 views • 24 slides
Cycle-Consistent Adversarial Learning as Approximate Bayesian Inference Louis C. Tiao 1 Edwin V.

Cycle-Consistent Adversarial Learning as Approximate Bayesian Inference Louis C. Tiao 1 Edwin V.

Cycle-Consistent Adversarial Learning as Approximate Bayesian Inference Louis C. Tiao 1 Edwin V. Bonilla 2 Fabio Ramos 1 July 22, 2018 1 University of Sydney, 2 University of New South Wales Motivation: Unpaired Image-to-Image Translation

329 views • 22 slides
Approximate Homomorphic Encryption and Privacy Preserving Machine Learning Jung Hee Cheon (SNU,

Approximate Homomorphic Encryption and Privacy Preserving Machine Learning Jung Hee Cheon (SNU,

Approximate Homomorphic Encryption and Privacy Preserving Machine Learning Jung Hee Cheon (SNU, CryptoLab) Thanks to YongSoo Song, Kiwoo Lee, Andrey KIM Outline 1. Homomorphic Encryption 2. HEAAN 3. Bootstrapping of HEAAN 4. Toolkit for

981 views • 50 slides
Learning in extended and approximate Rational Speech Acts models Christopher Potts Stanford

Learning in extended and approximate Rational Speech Acts models Christopher Potts Stanford

A Gricean ideal Implicatures RSA Learned RSA Neural RSA Language and action Prospects Learning in extended and approximate Rational Speech Acts models Christopher Potts Stanford Linguistics EMNLP 2016 Will Monroe 1 / 56 A Gricean ideal

2.04k views • 162 slides
Learning as Search Optimization: Approximate Large Margin Methods for Structured Prediction Hal

Learning as Search Optimization: Approximate Large Margin Methods for Structured Prediction Hal

Hal Daum III (hdaume@isi.edu) Learning as Search Optimization: Approximate Large Margin Methods for Structured Prediction Hal Daum III and Daniel Marcu Information Sciences Institute University of Southern California {hdaume,marcu}@isi.edu

400 views • 20 slides
Q Learning Forall s, a Initialize Q(s, a) = 0 Repeat Forever Where are you? s. Choose some

Q Learning Forall s, a Initialize Q(s, a) = 0 Repeat Forever Where are you? s. Choose some

11/9/16 Approximate Q-Learning Dan Weld / University of Washington [Many slides taken from Dan Klein and Pieter Abbeel / CS188 Intro to AI at UC Berkeley materials available at http://ai.berkeley.edu.] Q Learning Forall s, a Initialize

523 views • 20 slides
Approximate inference: Sampling methods Probabilistic Graphical Models Sharif University of

Approximate inference: Sampling methods Probabilistic Graphical Models Sharif University of

Approximate inference: Sampling methods Probabilistic Graphical Models Sharif University of Technology Spring 2018 Soleymani Approximate inference Approximate inference techniques Deterministic approximation Variational algorithms

1.96k views • 97 slides
Two Approximate- Programmability Birds, One Statistical- Inference Stone Adrian Sampson

Two Approximate- Programmability Birds, One Statistical- Inference Stone Adrian Sampson

Two Approximate- Programmability Birds, One Statistical- Inference Stone Adrian Sampson University of Washington sa pa APPROX 2014 Assisted approximate programming Statistical inference Cheap check generation 1 Assisted approximate

668 views • 25 slides
Learning with Approximate Kernel Embeddings Dino Sejdinovic Department of Statistics University

Learning with Approximate Kernel Embeddings Dino Sejdinovic Department of Statistics University

Learning with Approximate Kernel Embeddings Dino Sejdinovic Department of Statistics University of Oxford RegML Workshop, Simula, Oslo, 06/05/2017 D.Sejdinovic (University of Oxford) Learning with Kernel Embeddings Oslo, 06/05/2017 1 / 18

286 views • 24 slides
Approximate Dynamic Programming A. LAZARIC ( SequeL Team @INRIA-Lille ) ENS Cachan - Master 2 MVA

Approximate Dynamic Programming A. LAZARIC ( SequeL Team @INRIA-Lille ) ENS Cachan - Master 2 MVA

Approximate Dynamic Programming A. LAZARIC ( SequeL Team @INRIA-Lille ) ENS Cachan - Master 2 MVA SequeL INRIA Lille MVA-RL Course Approximate Dynamic Programming (a.k.a. Batch Reinforcement Learning) A. LAZARIC Reinforcement Learning

1.74k views • 140 slides
Empirical Comparison of Approximate Inference Algorithms for Networked Data Prithviraj Sen Lise

Empirical Comparison of Approximate Inference Algorithms for Networked Data Prithviraj Sen Lise

Empirical Comparison of Approximate Inference Algorithms for Networked Data Prithviraj Sen Lise Getoor Department of Computer Science University of Maryland, College Park. Workshop on Open Problems in Statistical Relational Learning, 2006

326 views • 16 slides
Approximate Reasoning for the Semantic Web Part V Approximate Resolution for OWL Frank van

Approximate Reasoning for the Semantic Web Part V Approximate Resolution for OWL Frank van

van Harmelen, Hitzler, Wache ESSLLI 2006 Malaga, Spain August 2006 AIFB Approximate Reasoning for the Semantic Web Part V Approximate Resolution for OWL Frank van Harmelen Pascal Hitzler Holger Wache ESSLLI 2006 Summer School

542 views • 25 slides

More recommend