safe exploration for interactive machine learning
play

Safe Exploration for Interactive Machine Learning Matteo Turchetta, - PowerPoint PPT Presentation

Aug 05, 2023 •147 likes •285 views

Safe Exploration for Interactive Machine Learning Matteo Turchetta, Felix Berkenkamp, Andreas Krause <latexit

  1. Safe Exploration for Interactive Machine Learning Matteo Turchetta, Felix Berkenkamp, Andreas Krause

  2. <latexit sha1_base64="6dFmwAauOuGIL1koQpOLW+AHK4o=">AB7XicbVBNSwMxEJ2tX7V+VT16CRahIpTdKuix6MVjBfsB7VKyabaNZpMlyapl6X/w4kERr/4fb/4b03YP2vpg4PHeDPzgpgzbVz328ktLa+sruXCxubW9s7xd29paJIrRBJeqHWBNORO0YZjhtB0riqOA01ZwfzXxWw9UaSbFrRnF1I/wQLCQEWys1AzLT8cnj71iya24U6BF4mWkBnqveJXty9JElFhCMdadzw3Nn6KlWGE03Ghm2gaY3KPB7RjqcAR1X46vXaMjqzSR6FUtoRBU/X3RIojrUdRYDsjbIZ63puI/3mdxIQXfspEnBgqyGxRmHBkJq8jvpMUWL4yBJMFLO3IjLEChNjAyrYELz5lxdJs1rxTivVm7NS7TKLIw8HcAhl8OAcanANdWgAgTt4hld4c6Tz4rw7H7PWnJPN7MfOJ8/r6Oiw=</latexit> <latexit sha1_base64="hL+FaLtOT9luwfLW3Ut08xl3Pcw=">AB6HicbVDLTgJBEOzF+IL9ehlIjHxRHbRI9ELx4hkUcCGzI79MLI7OxmZtZICF/gxYPGePWTvPk3DrAHBSvpFLVne6uIBFcG9f9dnJr6xubW/ntws7u3v5B8fCoqeNUMWywWMSqHVCNgktsG4EthOFNAoEtoLR7cxvPaLSPJb3ZpygH9GB5CFn1Fip/tQrltyOwdZJV5GSpCh1it+dfsxSyOUhgmqdcdzE+NPqDKcCZwWuqnGhLIRHWDHUkj1P5kfuiUnFmlT8JY2ZKGzNXfExMaT2OAtsZUTPUy95M/M/rpCa89idcJqlByRaLwlQE5PZ16TPFTIjxpZQpri9lbAhVZQZm03BhuAtv7xKmpWyd1Gu1C9L1ZsjycwCmcgwdXUIU7qEDGCA8wyu8OQ/Oi/PufCxac042cwx/4Hz+AOeHjQA=</latexit> Interactive Machine Learning • Agent can query no noisy sy values of an unk unkno nown n function • Use data to make inform rmed queries x f ( x ) + w • Available queries may depend from previous ones: model dependency with directed graph • Includes: Bayesian optimization, active learning and exploration of deterministic Markov decision processes Icon made by Freepik, Good Ware from www.flaticon.com Matteo Turchetta

  3. <latexit sha1_base64="hL+FaLtOT9luwfLW3Ut08xl3Pcw=">AB6HicbVDLTgJBEOzF+IL9ehlIjHxRHbRI9ELx4hkUcCGzI79MLI7OxmZtZICF/gxYPGePWTvPk3DrAHBSvpFLVne6uIBFcG9f9dnJr6xubW/ntws7u3v5B8fCoqeNUMWywWMSqHVCNgktsG4EthOFNAoEtoLR7cxvPaLSPJb3ZpygH9GB5CFn1Fip/tQrltyOwdZJV5GSpCh1it+dfsxSyOUhgmqdcdzE+NPqDKcCZwWuqnGhLIRHWDHUkj1P5kfuiUnFmlT8JY2ZKGzNXfExMaT2OAtsZUTPUy95M/M/rpCa89idcJqlByRaLwlQE5PZ16TPFTIjxpZQpri9lbAhVZQZm03BhuAtv7xKmpWyd1Gu1C9L1ZsjycwCmcgwdXUIU7qEDGCA8wyu8OQ/Oi/PufCxac042cwx/4Hz+AOeHjQA=</latexit> <latexit sha1_base64="CtSaWxXLG+cQl1A/lMlyUODJ8dM=">AB7nicbVA9SwNBEJ2LXzF+RS1tFoMQm3AXBS0sgjaWEcwHJEfY2+wlS/Z2z909MRz5ETYWitj6e+z8N26SKzTxwcDjvRlm5gUxZ9q47reTW1ldW9/Ibxa2tnd294r7B0tE0Vog0guVTvAmnImaMw2k7VhRHAaetYHQz9VuPVGkmxb0Zx9SP8ECwkBFsrNR6KD+dXiG3Vy5FXcGtEy8jJQgQ71X/Or2JUkiKgzhWOuO58bGT7EyjHA6KXQTWNMRnhAO5YKHFHtp7NzJ+jEKn0USmVLGDRTf0+kONJ6HAW2M8JmqBe9qfif10lMeOmnTMSJoYLMF4UJR0ai6e+ozxQlho8twUQxeysiQ6wMTahg3BW3x5mTSrFe+sUr07L9WuszjycATHUAYPLqAGt1CHBhAYwTO8wpsTOy/Ou/Mxb8052cwh/IHz+QPFw46K</latexit> <latexit sha1_base64="1WDpB7YDdQ1gSi87g0UxOTPeTV4=">AB8XicbVDLSgNBEOz1GeMr6tHLYBDiJexGQY9BLx4jmAcmS5idJIhs7ObmVkxLPkLx4U8erfePNvnDwOmljQUFR1090VxIJr47rfzsrq2vrGZmYru72zu7efOzis6ShRDKsEpFqBFSj4BKrhuBjVghDQOB9WBwM/Hrj6g0j+S9GcXoh7QneZczaqz0MCw8nbV6OCRuO5d3i+4UZJl4c5KHOSrt3FerE7EkRGmYoFo3PTc2fkqV4UzgONtKNMaUDWgPm5ZKGqL20+nFY3JqlQ7pRsqWNGSq/p5Iaj1KAxsZ0hNXy96E/E/r5mY7pWfchknBiWbLeomgpiITN4nHa6QGTGyhDLF7a2E9amizNiQsjYEb/HlZVIrFb3zYunuIl+nseRgWM4gQJ4cAluIUKVIGBhGd4hTdHOy/Ou/Mxa1x5jNH8AfO5w9OaZAF</latexit> <latexit sha1_base64="6dFmwAauOuGIL1koQpOLW+AHK4o=">AB7XicbVBNSwMxEJ2tX7V+VT16CRahIpTdKuix6MVjBfsB7VKyabaNZpMlyapl6X/w4kERr/4fb/4b03YP2vpg4PHeDPzgpgzbVz328ktLa+sruXCxubW9s7xd29paJIrRBJeqHWBNORO0YZjhtB0riqOA01ZwfzXxWw9UaSbFrRnF1I/wQLCQEWys1AzLT8cnj71iya24U6BF4mWkBnqveJXty9JElFhCMdadzw3Nn6KlWGE03Ghm2gaY3KPB7RjqcAR1X46vXaMjqzSR6FUtoRBU/X3RIojrUdRYDsjbIZ63puI/3mdxIQXfspEnBgqyGxRmHBkJq8jvpMUWL4yBJMFLO3IjLEChNjAyrYELz5lxdJs1rxTivVm7NS7TKLIw8HcAhl8OAcanANdWgAgTt4hld4c6Tz4rw7H7PWnJPN7MfOJ8/r6Oiw=</latexit> Safety constrained interactive machine learning f ( x ) + w q ( x ) ≥ 0 Unknown safety constrai raint q(x)>0 that must be satisfied at al all ti times x q ( x ) < 0 Encompasses many problems Therapy design Mars exploration Model free RL [Sui et al. 2015], [Turchetta et al. 2016], [Berkenkamp et al. 2016] [Sui et al. 2018] [Wachi et al. 2018] Icon made by Smashicons from www.flaticon.com Matteo Turchetta

  4. <latexit sha1_base64="vhcSZKB5REty1fMyhkxMk/A5FwM=">AB8nicbVDLSgMxFL1TX7W+qi7dBIvgqsxUQZdFXbisYB8wHUomzbShmWRIMkIZ+hluXCji1q9x59+YaWehrQcCh3PuJeMOFMG9f9dkpr6xubW+Xtys7u3v5B9fCo2WqCG0TyaXqhVhTzgRtG2Y47SWK4jktBtObnO/+0SVZlI8mlCgxiPBIsYwcZKfj/GZkwz+5mg2rNrbtzoFXiFaQGBVqD6ld/KEkaU2EIx1r7npuYIMPKMLprNJPNU0wmeAR9S0VOKY6yOaRZ+jMKkMUSWfMGiu/t7IcKz1NA7tZB5RL3u5+J/npya6DjImktRQRYfRSlHRqL8fjRkihLDp5ZgopjNisgYK0yMbaliS/CWT14lnUbdu6g3Hi5rzZuijKcwCmcgwdX0IR7aEbCEh4hld4c4z4rw7H4vRklPsHMfOJ8/diuRXg=</latexit> <latexit sha1_base64="PkvxSbIJAC9+r2/QYT+PqTqc0Ag=">AB6nicbVBNS8NAEJ34WetX1aOXxSJ4KkV9Fj0oMeK9gPaUDbTbt0swm7E6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmjVjDdYLGPdDqjhUijeQIGStxPNaRI3gpGN1O/9cS1EbF6xHC/YgOlAgFo2ilh9se9kplt+LOQJaJl5My5Kj3Sl/dfszSiCtkhrT8dwE/YxqFEzySbGbGp5QNqID3rFU0YgbP5udOiGnVumTMNa2FJKZ+nsio5Ex4yiwnRHFoVn0puJ/XifF8MrPhEpS5IrNF4WpJBiT6d+kLzRnKMeWUKaFvZWwIdWUoU2naEPwFl9eJs1qxTuvVO8vyrXrPI4CHMJnIEHl1CDO6hDAxgM4Ble4c2Rzovz7nzMW1ecfOYI/sD5/AEn5I2</latexit> <latexit sha1_base64="PkvxSbIJAC9+r2/QYT+PqTqc0Ag=">AB6nicbVBNS8NAEJ34WetX1aOXxSJ4KkV9Fj0oMeK9gPaUDbTbt0swm7E6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmjVjDdYLGPdDqjhUijeQIGStxPNaRI3gpGN1O/9cS1EbF6xHC/YgOlAgFo2ilh9se9kplt+LOQJaJl5My5Kj3Sl/dfszSiCtkhrT8dwE/YxqFEzySbGbGp5QNqID3rFU0YgbP5udOiGnVumTMNa2FJKZ+nsio5Ex4yiwnRHFoVn0puJ/XifF8MrPhEpS5IrNF4WpJBiT6d+kLzRnKMeWUKaFvZWwIdWUoU2naEPwFl9eJs1qxTuvVO8vyrXrPI4CHMJnIEHl1CDO6hDAxgM4Ble4c2Rzovz7nzMW1ecfOYI/sD5/AEn5I2</latexit> <latexit sha1_base64="iTYg1Ft2L5xMtjPOc3OrWXcTUmE=">AB+XicbVDLSsNAFJ3UV62vqEs3wSK4KkVdFl047KifUAbw2Q6aYdOZsLMTaGE/okbF4q49U/c+TdO2iy09cDA4Zx7uHdOmHCmwXW/rdLa+sbmVnm7srO7t39gHx61tUwVoS0iuVTdEGvKmaAtYMBpN1EUxyGnXB8m/udCVWaSfEI04T6MR4KFjGCwUiBbfelsfN09jB7SgI7Kpbc+dwVolXkCoq0Azsr/5AkjSmAgjHWvc8NwE/woY4XRW6aeaJpiM8ZD2DBU4ptrP5pfPnDOjDJxIKvMEOHP1dyLDsdbTODSTMYaRXvZy8T+vl0J07WdMJClQRaLopQ7IJ28BmfAFCXAp4Zgopi51SEjrDABU1bFlOAtf3mVtOs176JWv7+sNm6KOsroBJ2ic+ShK9RAd6iJWoigCXpGr+jNyqwX6936WIyWrCJzjP7A+vwBIxyT+w=</latexit> <latexit sha1_base64="iTYg1Ft2L5xMtjPOc3OrWXcTUmE=">AB+XicbVDLSsNAFJ3UV62vqEs3wSK4KkVdFl047KifUAbw2Q6aYdOZsLMTaGE/okbF4q49U/c+TdO2iy09cDA4Zx7uHdOmHCmwXW/rdLa+sbmVnm7srO7t39gHx61tUwVoS0iuVTdEGvKmaAtYMBpN1EUxyGnXB8m/udCVWaSfEI04T6MR4KFjGCwUiBbfelsfN09jB7SgI7Kpbc+dwVolXkCoq0Azsr/5AkjSmAgjHWvc8NwE/woY4XRW6aeaJpiM8ZD2DBU4ptrP5pfPnDOjDJxIKvMEOHP1dyLDsdbTODSTMYaRXvZy8T+vl0J07WdMJClQRaLopQ7IJ28BmfAFCXAp4Zgopi51SEjrDABU1bFlOAtf3mVtOs176JWv7+sNm6KOsroBJ2ic+ShK9RAd6iJWoigCXpGr+jNyqwX6936WIyWrCJzjP7A+vwBIxyT+w=</latexit> Existing approaches D p Build a conserv rvat ative estimat ate of the decisions that are safe to evaluate S t p S t Uni Uniforml mly reduce uce unce uncertaint nty on the boundary of this region G t G t Treating the ex expansion of the safe set as a pro proxy xy objective can be was waste teful Example: 1D optimization task StageOPT [Sui et al. 2018] Many unnecessary samples when optimum has already been found f ( s ) ≡ q ( s ) q ( s ) ≥ 0 # evaluations Domain D Matteo Turchetta

Recommend

Machine Learning - Intro Aarti Singh Machine Learning 10-701/15-781 Sept 8, 2010 You tell me

Machine Learning - Intro Aarti Singh Machine Learning 10-701/15-781 Sept 8, 2010 You tell me

Machine Learning - Intro Aarti Singh Machine Learning 10-701/15-781 Sept 8, 2010 You tell me This class is going to be interactive! What is Machine Learning? 2 What is Machine Learning? 3 What is Machine Learning? Study of

850 views • 52 slides
Interactive Machine Learning via Transparent Modeling: Putting Human Experts in the Drivers

Interactive Machine Learning via Transparent Modeling: Putting Human Experts in the Drivers

Interactive Machine Learning via Transparent Modeling: Putting Human Experts in the Drivers Seat Rich Caruana Microsoft Research Joint Work with Sarah Tan &amp; Yin Lou Johannes Gehrke, Paul Koch, Marc Sturm, Noemie Elhadad Thanks to Greg

842 views • 82 slides
An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning http://www.cs.iastate.edu/~cs573x/bbsilab.html Machine Learning Software Preparing Data Building Classifiers Interpreting Results Machine Learning Software

496 views • 26 slides
Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by

Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by

Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by which computers can be trained through observation, rather than being explicitly programmed. Basically, a program is given input and adjusts its

556 views • 17 slides
The What-If Tool (WIT) Interactive Probing of Machine Learning Models James Wexler, Mahima

The What-If Tool (WIT) Interactive Probing of Machine Learning Models James Wexler, Mahima

The What-If Tool (WIT) Interactive Probing of Machine Learning Models James Wexler, Mahima Pushkarna, Tolga Bolukbasi, Martin Wattenberg, Fernanda Viegas, and Jimbo Wilson Presented on Nov 19, by Patrick Huber Problem &amp; Objective Problem:

452 views • 23 slides
Machine Learning: Study of algorithms that improve their performance P at some task T

Machine Learning: Study of algorithms that improve their performance P at some task T

Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 12, 2015 Today: Readings: What is machine learning? The Discipline of ML Decision tree learning Mitchell, Chapter 3

872 views • 29 slides
Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning

Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning

GCT634/AI613: Musical Applications of Machine Learning (Fall 2020) Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning Pipeline in Classification Tasks A set of hand-designed audio features are selected

398 views • 26 slides
The Human Experience in Interactive Machine Learning Karen M. Feigh Samantha Krening GOAL To

The Human Experience in Interactive Machine Learning Karen M. Feigh Samantha Krening GOAL To

The Human Experience in Interactive Machine Learning Karen M. Feigh Samantha Krening GOAL To enable people to naturally and intuitively teach agents to perform tasks. Widespread integration of robotics requires ML agents that are more

776 views • 34 slides
CPSC 340: Machine Learning and Data Mining Data Exploration Summer 2020 This lecture roughly

CPSC 340: Machine Learning and Data Mining Data Exploration Summer 2020 This lecture roughly

CPSC 340: Machine Learning and Data Mining Data Exploration Summer 2020 This lecture roughly follow: http://www-users.cs.umn.edu/~kumar/dmbook/dmslides/chap2_data.pdf Data Mining: Birds Eye View 1) Collect data. 2) Data mining! 3)

1.16k views • 77 slides
CS 335 Machine Learning What is Machine Learning? Dan Sheldon Spring 2019 What is Machine

CS 335 Machine Learning What is Machine Learning? Dan Sheldon Spring 2019 What is Machine

1/22/19 CS 335 Machine Learning What is Machine Learning? Dan Sheldon Spring 2019 What is Machine Learning? A Simple Task: Recognize Obama How do you program a computer to Recognize faces? Recommend movies? Decide which web

581 views • 5 slides
Machine Learning Machine Learning: algorithms that use experience to improve their

Machine Learning Machine Learning: algorithms that use experience to improve their

Machine Learning Machine Learning: algorithms that use experience to improve their performance We use machine learning in situations where it is very challenging (or impossible) to define the rules by hand: e.g. face detection

1.04k views • 28 slides
Applied Machine Learning CIML Chaps 4-5 (A Geometric Approach) A ship in port is safe, but

Applied Machine Learning CIML Chaps 4-5 (A Geometric Approach) A ship in port is safe, but

Applied Machine Learning CIML Chaps 4-5 (A Geometric Approach) A ship in port is safe, but that is not what ships are for. Grace Hopper (1906-1992) Week 3: Extensions and Variations of Perceptron; Practical Issues and HW1 Professor

494 views • 33 slides
Applied Machine Learning CIML Chaps 4-5 (A Geometric Approach) A ship in port is safe, but

Applied Machine Learning CIML Chaps 4-5 (A Geometric Approach) A ship in port is safe, but

Applied Machine Learning CIML Chaps 4-5 (A Geometric Approach) A ship in port is safe, but that is not what ships are for. Grace Hopper (1906-1992) Week 5: Extensions and Variations of Perceptron, and Practical Issues

328 views • 29 slides
Detecting and Characterizing Events Unsupervised Machine Learning for Social Science Allison

Detecting and Characterizing Events Unsupervised Machine Learning for Social Science Allison

Detecting and Characterizing Events Unsupervised Machine Learning for Social Science Allison J.B. Chaney Princeton University supervised learning unsupervised learning methods input output prediction exploration goals 9 3 1 ? 5 6

920 views • 59 slides
Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Introduction to Machine Learning 1 / 12 Outline 1 Nearest Neighbor Prediction 2 Complexity Considerations 3 The Voronoi Diagram 4 Overfitting

564 views • 12 slides
1 Why Study Machine Learning? Why Study Machine Learning? Cognitive Science The Time is Ripe

1 Why Study Machine Learning? Why Study Machine Learning? Cognitive Science The Time is Ripe

What is Learning? Herbert Simon: Learning is any process by which a system improves performance from CS 391L: Machine Learning experience. Introduction What is the task? Classification Problem solving / planning /

603 views • 6 slides
MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING

MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING

JACOPO DI IORIO, MATTEO FONTANA, DANIELE OCCHIUTO, CLAUDIA VOLPETTI MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING STATISTICAL LEARNING Separate evolutions, but with shared methodologies MERGE

565 views • 55 slides
Safe Exploration for Optimization with Gaussian Processes Yanan Sui Alkis Gotovos Joel W.

Safe Exploration for Optimization with Gaussian Processes Yanan Sui Alkis Gotovos Joel W.

Safe Exploration for Optimization with Gaussian Processes Yanan Sui Alkis Gotovos Joel W. Burdick Andreas Krause Caltech ETH Zurich Caltech ETH Zurich Better safe than sorry youtube.com/user/mattessons Safe Exploration for Optimization

922 views • 80 slides
Safe Machine Learning Silvia Chiappa &amp; Jan Leike ICML 2019 ML Research Reality horns

Safe Machine Learning Silvia Chiappa &amp; Jan Leike ICML 2019 ML Research Reality horns

Safe Machine Learning Silvia Chiappa &amp; Jan Leike ICML 2019 ML Research Reality horns offline datasets nose annotated a long time ago tail simulated environments abstract domains also more cute restart experiments at will ...

890 views • 76 slides
Bandits and Exploration: How do we (optimally) gather information? Sham M. Kakade Machine

Bandits and Exploration: How do we (optimally) gather information? Sham M. Kakade Machine

Bandits and Exploration: How do we (optimally) gather information? Sham M. Kakade Machine Learning for Big Data CSE547/STAT548 University of Washington S. M. Kakade (UW) Optimization for Big data 1 / 18 Announcements... HW 4 posted soon

710 views • 24 slides
Apache PredictionIO End-to-End Machine Learning Server with Apache Spark What is Machine

Apache PredictionIO End-to-End Machine Learning Server with Apache Spark What is Machine

Apache PredictionIO End-to-End Machine Learning Server with Apache Spark What is Machine Learning? What is Machine Learning? Examples of machine learning applications: Facial recognition software E-mail spam detectors Automated

302 views • 17 slides
Machine Learning 11 AI Slides (6e) c Lin Zuoquan@PKU 1998-2020 11 1 11 Machine Learning

Machine Learning 11 AI Slides (6e) c Lin Zuoquan@PKU 1998-2020 11 1 11 Machine Learning

Machine Learning 11 AI Slides (6e) c Lin Zuoquan@PKU 1998-2020 11 1 11 Machine Learning 11.1 Learning agents 11.2 Inductive learning 11.3 Deep learning 11.4 Statistical learning 11.5 Reinforcement learning 11.6 Transfer learning

1.7k views • 159 slides
Deep Learning: Intro Juhan Nam Review of Traditional Machine Learning The traditional machine

Deep Learning: Intro Juhan Nam Review of Traditional Machine Learning The traditional machine

GCT634/AI613: Musical Applications of Machine Learning (Fall 2020) Deep Learning: Intro Juhan Nam Review of Traditional Machine Learning The traditional machine learning pipeline Temporal Frame-level Unsupervised Classifier Summary

613 views • 27 slides
Machine Learning for Auto Optimization What is Machine Learning? Definition: Machine

Machine Learning for Auto Optimization What is Machine Learning? Definition: Machine

Machine Learning for Auto Optimization What is Machine Learning? Definition: Machine learning refers to any system where the performance of a machine in performing a task improves by gaining more experience in performing that task .

394 views • 27 slides

More recommend