a hybrid reinforcement learning approach to autonomic
play

A Hybrid Reinforcement Learning Approach to Autonomic Resource - PowerPoint PPT Presentation

Jan 12, 2023 •173 likes •288 views

A Hybrid Reinforcement Learning Approach to Autonomic Resource Allocation (2006) Paper: Gerald Tesauro, Nicholas K. Jongt, Rajarshi Das and Mohamed N. Bennanit Presentation: Nat McAleese (nm583) What is autonomic computing? computing systems

  1. A Hybrid Reinforcement Learning Approach to Autonomic Resource Allocation (2006) Paper: Gerald Tesauro, Nicholas K. Jongt, Rajarshi Das and Mohamed N. Bennanit Presentation: Nat McAleese (nm583)

  2. What is autonomic computing? “ computing systems that can manage themselves given high-level objectives from administrators ” An IBM marketing buzzword? (probably) [0] Totally inevitable due to system complexity? Elastic scaling? [0] Kephart, Jeffrey O., and David M. Chess. "The vision of autonomic computing." Computer 36.1 (2003): 41-50.

  3. Problem Maximise system-wide utility, which is assumed to be based on system performance relative to SLAs (Service Level Objectives) Considers the sum of utilities from application specific utility functions. For example web services are penalised if they fall below some latency, batch jobs improve linearly with more added resources

  4. The scheme

  5. Critical Detail “Given enough training samples, RL can converge to the correct value function V π associated with any fixed policy π, and that the new policy whose behavior greedily maximizes V π is guaranteed to improve upon the original policy.” [0] [0] G. Tesauro et al.: A Hybrid Reinforcement Learning Approach to Autonomic Resource Allocation, ICAC, 2006.

  6. The scheme Each application manager submits a utility function, computed by evaluating the learnt Q(s, a) for all a. The resource arbiter then optimises [0] But it involves solving an NP-hard mixed integer programming model at the resource arbiter. Could it help with adding/removing processes? Is there an (unintentionally) adversarial component? Need more data. [0] Walsh, William E., et al. "Utility functions in autonomic systems." Autonomic Computing, 2004. Proceedings. International Conference on. IEEE, 2004.

  7. The network 3 inputs - current demand λ, n t-1 , n t 12 sigmoid hidden units 1 output. It’s tiny. [0] G. Tesauro et al.: A Hybrid Reinforcement Learning Approach to Autonomic Resource Allocation, ICAC, 2006.

  8. Results [0] G. Tesauro et al.: A Hybrid Reinforcement Learning Approach to Autonomic Resource Allocation, ICAC, 2006.

  9. (With the benefit of hindsight, paper was 2006) Future work? These approaches could be improved by *loads* of recent developments. I’d start with applying some (or all) of the recently discovered optimisations for reinforcement learning [0]: And recent work to improve the raw performance Of fully connected nets [1] The real problem is (in my opinion) is lack of a standard benchmark. [0] Hessel, Matteo, et al. "Rainbow: Combining Improvements in Deep Reinforcement Learning." arXiv preprint arXiv:1710.02298 (2017). [1] Klambauer, Günter, et al. "Self-Normalizing Neural Networks." arXiv preprint arXiv:1706.02515 (2017).

  10. (With the benefit of hindsight, paper was 2006) Criticism Does the system design make any sense at all? Is it more scalable? Their previous paper does not present a coherent argument for why this scheme is useful with *learned* utility functions, instead arguing that they are intuitive to design. Compatibility with software that’s actually used - OpenStack is becoming increasingly popular, and has been the target of similar work. Standard benchmarks Standard benchmarks.

  11. (With the benefit of hindsight, paper was 2006) Criticism From next week, the papers that lack a standard benchmarking approach: Dutreilh, Xavier, et al. "Using reinforcement learning for autonomic resource allocation in clouds: towards a fully automated workflow." ICAS 2011, The Seventh International Conference on Autonomic and Autonomous Systems. 2011. LaCurts, Katrina, et al. "Cicada: Introducing Predictive Guarantees for Cloud Networks." HotCloud 14 (2014): 14-19. Roy, Nilabja, Abhishek Dubey, and Aniruddha Gokhale. "Efficient autoscaling in the cloud using predictive models for workload forecasting." Cloud Computing (CLOUD), 2011 IEEE International Conference on. IEEE, 2011. Delimitrou, Christina, and Christos Kozyrakis. "Quasar: resource-efficient and QoS-aware cluster management." ACM SIGPLAN Notices. Vol. 49. No. 4. ACM, 2014. The CherryPick guys tried though! - used, SparkKM, SparkReg TPC-DS, TPC-H, TeraSort. Unfortunately these are just batch processing workloads. - Alipourfard, Omid, et al. "CherryPick: Adaptively Unearthing the Best Cloud Configurations for Big Data Analytics." NSDI. 2017.

Recommend

Hybrid System Falsification and Reinforcement Learning Formal Method for Cyber-Physical Systems

Hybrid System Falsification and Reinforcement Learning Formal Method for Cyber-Physical Systems

Hybrid System Falsification and Reinforcement Learning Formal Method for Cyber-Physical Systems Clovis Eberhart David Sprunger National Institute of Technology, Japan SOKENDAI lesson, July 1, 8, and 22 1 / 50 Lecture structure Lectures

1.01k views • 83 slides
RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

551 views • 35 slides
Hybrid System Falsification and Reinforcement Learning Formal Method for Cyber-Physical Systems

Hybrid System Falsification and Reinforcement Learning Formal Method for Cyber-Physical Systems

Hybrid System Falsification and Reinforcement Learning Formal Method for Cyber-Physical Systems Clovis Eberhart David Sprunger National Institute of Technology, Japan SOKENDAI lesson, July 1, 8, and 22 1 / 31 Quick reminder Falsification:

405 views • 36 slides
Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine playing a new game whose rules you dont know; after a hundred or so moves your don t know; after a hundred or so moves, your opponent announces, You

511 views • 30 slides
CULE*: GPU ACCELERATED RL * CUDA Learning Environment Reinforcement Learning Steven Dalton, Iuri

CULE*: GPU ACCELERATED RL * CUDA Learning Environment Reinforcement Learning Steven Dalton, Iuri

March 26-29, 2018 | Silicon Valley CULE*: GPU ACCELERATED RL * CUDA Learning Environment Reinforcement Learning Steven Dalton, Iuri Frosio, Jared Hoberock, Jason Clemons time REINFORCEMENT LEARNING A successful approach Board games

551 views • 41 slides
Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers

Reinforcement Learning You can think of supervised learning as the teacher providing answers (the class labels) Reinforcement learning: In reinforcement learning, the agent learns Markov Decision Processes based on a punishment/reward

89 views • 6 slides
Examples of Reinforcement Learning Robocup Soccer Teams Stone & Veloso, Reidmiller et al.

Examples of Reinforcement Learning Robocup Soccer Teams Stone & Veloso, Reidmiller et al.

Reinforcement Learning Slides by Rich Sutton Mods by Dan Lizotte Refer to Reinforcement Learning: An Introduction by Sutton and Barto Alpaydin Chapter 16 Up until now we have been Supervised Learning Classifying, mostly

891 views • 31 slides
A Bayesian Approach to Finding Compact Representations for Reinforcement Learning Special thanks

A Bayesian Approach to Finding Compact Representations for Reinforcement Learning Special thanks

A Bayesian Approach to Finding Compact Representations for Reinforcement Learning Special thanks to Joelle Pineau for presenting our paper - July, 2012 1 Authors Alborz Geramifard Stefanie Tellex David Wingate Nicholas Roy Jonathan How 2

491 views • 25 slides
CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning

CS885 Reinforcement Learning Module 2: June 6, 2020 Maximum Entropy Reinforcement Learning Haarnoja, Tang et al. (2017) Reinforcement Learning with Deep Energy Based Policies, ICML . Haarnoja, Zhou et al. (2018) Soft Actor-Critic: Off-Policy

683 views • 24 slides
Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control

Reinforcement Learning Reinforcement Learning Now that you know a little about Optimal Control Theory, you actually have some knowledge in RL. RL shares the overall goal with OCT: solving for a control policy such that the cumulative cost

610 views • 22 slides
Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning II Still assume a Marko kov decision process (MDP): A se set o of st states s s S A se set o of a actions ( s (per st state) A A mo model

96 views • 5 slides
R i f R i f Reinforcement Learning III Reinforcement Learning III t L t L i i III III Dec

R i f R i f Reinforcement Learning III Reinforcement Learning III t L t L i i III III Dec

R i f R i f Reinforcement Learning III Reinforcement Learning III t L t L i i III III Dec 03 2008 1 Large State Spaces h When a problem has a large state space we can not longer represent the U or Q functions as explicit tables explicit

436 views • 18 slides
Reinforcement Learning UMaine COS 470/570 Introduction to AI Why reinforcement learning?

Reinforcement Learning UMaine COS 470/570 Introduction to AI Why reinforcement learning?

Reinforcement Learning<br/><br/> 4/25/19, 8*06 PM Reinforcement Learning<br/><br/> 4/25/19, 8*06 PM Reinforcement Learning UMaine COS 470/570 Introduction to AI Why reinforcement learning? Spring 2019 Created:

371 views • 15 slides
7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B.

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B.

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B. Temporal Difference Reinforcement Learning C. PVLV Model D. Cerebellum and Error-driven Learning 2/23/18 COSC 494/594 CCN 2 Sensory-Motor Loop

790 views • 56 slides
Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

541 views • 15 slides
Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning Ja Jan-Wi

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning Ja Jan-Wi

Reinforcement Learning CS 4100: Artificial Intelligence Reinforcement Learning Ja Jan-Wi Willem van de Meent Northeastern University [These slides were created by Dan Klein and Pieter Abbeel for CS188 Intro to AI at UC Berkeley. All CS188

145 views • 10 slides
Reinforcement Learning AIMA Chapters: 21.1, 21.2, 21.3. Sutton and Barto, Reinforcement Learning:

Reinforcement Learning AIMA Chapters: 21.1, 21.2, 21.3. Sutton and Barto, Reinforcement Learning:

Reinforcement Learning Reinforcement Learning AIMA Chapters: 21.1, 21.2, 21.3. Sutton and Barto, Reinforcement Learning: an Introduction, 2nd Edition: Chapters 6 (6.1 6.5) Outline Reinforcement Learning Reinforcement Learning: the

587 views • 27 slides
Unpaired Sentiment-to-Sentiment Translation: A Cycled Reinforcement Learning Approach Jingjing

Unpaired Sentiment-to-Sentiment Translation: A Cycled Reinforcement Learning Approach Jingjing

Unpaired Sentiment-to-Sentiment Translation: A Cycled Reinforcement Learning Approach Jingjing Xu, Xu Sun, Qi Zeng, Xuancheng Ren, Xiaodong Zhang, Houfeng Wang, Wenjie Li MOE Key Lab of Computational Linguistics, School of EECS, Peking University

699 views • 34 slides
A Marketing Game: A Reinforcement Learning Approach to Optimizing Preference on a Social Network

A Marketing Game: A Reinforcement Learning Approach to Optimizing Preference on a Social Network

A Marketing Game: A Reinforcement Learning Approach to Optimizing Preference on a Social Network Matthew G. Reyes OReilly Applied AI April 18, 2018 Motivation and Contribution consumers choose between two alternatives, A and B Pepsi

630 views • 50 slides
Module 11 Introduction to Reinforcement Learning CS 886 Sequential Decision Making and

Module 11 Introduction to Reinforcement Learning CS 886 Sequential Decision Making and

Module 11 Introduction to Reinforcement Learning CS 886 Sequential Decision Making and Reinforcement Learning University of Waterloo Machine Learning Supervised Learning Teacher tells learner what to remember Reinforcement Learning

401 views • 29 slides
osu!mania Reinforcement Learning Agent

osu!mania Reinforcement Learning Agent

osu!mania Reinforcement Learning Agent ichrysomallis@isc.tuc.gr 2014030078 Contents Introduction osu!mania game Graphical User Interface customization Agents environment Approach and

136 views • 13 slides
1 Pure Reinforcement Learning vs. Reinforcement Learning Monte-Carlo Planning No knowledge

1 Pure Reinforcement Learning vs. Reinforcement Learning Monte-Carlo Planning No knowledge

CSE 573: Artificial Intelligence Todo Reinforcement Learning Add simulations from 473 Dan Weld Add UCB bound (cut bolzman & constant epsilon Add snazzy videos (pendulum, zico kolter See http://www See http://www-

462 views • 11 slides
Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest

Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest

Safe Reinforcement Learning Philip S. Thomas Stanford CS234: Reinforcement Learning, Guest Lecture May 24, 2017 Lecture overview What makes a reinforcement learning algorithm safe ? Notation Creating a safe reinforcement learning

1.42k views • 88 slides
Autonomic Dysfunction: Autonomic Non-Epileptic Seizures and the Autonomic Epilepsies James J.

Autonomic Dysfunction: Autonomic Non-Epileptic Seizures and the Autonomic Epilepsies James J.

Autonomic Dysfunction: Autonomic Non-Epileptic Seizures and the Autonomic Epilepsies James J. Riviello, Jr., MD Sergievsky Family Professor of Neurology and Pediatrics Director, Division of Child Neurology Department of Neurology Columbia

836 views • 58 slides

More recommend