learning to generalize from sparse and underspecified
play

Learning to Generalize from Sparse and Underspecified Rewards - PowerPoint PPT Presentation

Jul 14, 2023 •383 likes •571 views

Proprietary + Confidential Learning to Generalize from Sparse and Underspecified Rewards Rishabh Agarwal, Chen Liang, Dale Schuurmans, Mohammad Norouzi Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

  1. Proprietary + Confidential Learning to Generalize from Sparse and Underspecified Rewards Rishabh Agarwal, Chen Liang, Dale Schuurmans, Mohammad Norouzi Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

  2. Proprietary + Confidential Motivation Reinforcement learning has enabled remarkable advances: ➢ These advances hinge on the availability of high-quality and dense rewards. ➢ However, many real-world problems involve sparse and underspecified ➢ rewards. Language understanding tasks provide a natural way to investigate RL ➢ algorithms in such settings. Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

  3. Proprietary + Confidential Instruction Following Instruction: “Right Up Up Right” : Blindfolded agent : Goal : Death Possible Actions: ← , ↑ , → , ↓ The reward is +1 if the goal is reached and 0 otherwise. Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

  4. Proprietary + Confidential Weakly-supervised Semantic Parsing Question: Which nation won the most number of Silver medals? Rank Nation Gold Silver Bronze Total 1 Nigeria 13 16 9 38 ????? 2 Kenya 12 10 7 29 3 Ethiopia 4 3 4 11 ... ... ... ... ... ... 15 Madagascar 0 0 2 2 Nigeria Tanzania 0 0 1 1 16 Uganda 0 0 1 1 Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

  5. Proprietary + Confidential Challenges: (1) Exploration, (2) Generalization Question: Which nation won the most number of Silver medals? Rank Nation Gold Silver Bronze Total 1 Nigeria 13 16 9 38 ????? 2 Kenya 12 10 7 29 3 Ethiopia 4 3 4 11 ... ... ... ... ... ... 15 Madagascar 0 0 2 2 Nigeria Tanzania 0 0 1 1 16 Uganda 0 0 1 1 Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

  6. Proprietary + Confidential Underspecified Rewards Instruction: “Right Up Up Right” Correct Action Sequence: → ↑ ↑ → Spurious Action Sequences: ↑ → ↑ → ↑ → → ↑ ↑ → ↑ ↓ → ↑ Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

  7. Proprietary + Confidential Underspecified Rewards Question: Which nation won the most number of Silver medals? Rank Nation Gold Silver Bronze Total 1 Nigeria 13 16 9 38 v0 = (argmax all_rows r.Silver) 2 Kenya 12 10 7 29 return (hop v0 r.Nation) 3 Ethiopia 4 3 4 11 ... ... ... ... ... ... 15 Madagascar 0 0 2 2 Nigeria Tanzania 0 0 1 1 16 Uganda 0 0 1 1 Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

  8. Proprietary + Confidential Underspecified Rewards Question: Which nation won the most number of Silver medals? Rank Nation Gold Silver Bronze Total 1 Nigeria 13 16 9 38 v0 = (argmax all_rows r.Gold) 2 Kenya 12 10 7 29 return (hop v0 r.Nation) 3 Ethiopia 4 3 4 11 ... ... ... ... ... ... 15 Madagascar 0 0 2 2 Nigeria Tanzania 0 0 1 1 16 Uganda 0 0 1 1 Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

  9. Proprietary + Confidential Underspecified Rewards Question: Which nation won the most number of Silver medals? Rank Nation Gold Silver Bronze Total 1 Nigeria 13 16 9 38 v0 = (argmin all_rows r.Rank) 2 Kenya 12 10 7 29 return (hop v0 r.Nation) 3 Ethiopia 4 3 4 11 ... ... ... ... ... ... 15 Madagascar 0 0 2 2 Nigeria Tanzania 0 0 1 1 16 Uganda 0 0 1 1 Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

  10. Proprietary + Confidential Underspecified Rewards Awesome Reinforcement Learning Model Recent interest in automated reward learning using expert demonstrations. Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

  11. Proprietary + Confidential Learning Rewards without Demonstration Awesome Reinforcement Learning Model Recent interest in automated reward learning using expert demonstrations. What if we don’t have demonstrations? Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

  12. Proprietary + Confidential Learning Rewards without Demonstration Awesome Reinforcement Learning Model Recent interest in automated reward learning using expert demonstrations. Key idea: Use generalization error as the supervisory signal for learning rewards. Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

  13. Proprietary + Confidential Meta Reward Learning (MeRL) The auxiliary rewards R ϕ are optimized based on the generalization performance O val of a policy π ϴ trained using the auxiliary rewards: Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

  14. Proprietary + Confidential Tackling Sparse Rewards Disentangle exploration from exploitation. ➢ Mode covering direction of KL divergence to collect successful ➢ sequences . Mode seeking direction of KL divergence for robust optimization. ➢ Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

  15. Proprietary + Confidential Results MAPOX uses our mode ➢ Method WikiSQL WikiTable covering exploration strategy on top of prior work ( MAPO) . MAPO 72.4 ( ± 0.3) 42.9 ( ± 0.5) MAPOX 74.2 ( ± 0.4) 43.3 ( ± 0.4) Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

  16. Proprietary + Confidential Results MAPOX uses our mode ➢ Method WikiSQL WikiTable covering exploration strategy on top of prior work ( MAPO) . MAPO 72.4 ( ± 0.3) 42.9 ( ± 0.5) BoRL is our Bayesian ➢ MAPOX 74.2 ( ± 0.4) 43.3 ( ± 0.4) optimization approach for learning rewards. BoRL 74.2 ( ± 0.2) 43.8 ( ± 0.2) Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

  17. Proprietary + Confidential Results MAPOX uses our mode ➢ Method WikiSQL WikiTable covering exploration strategy on top of prior work ( MAPO) . MAPO 72.4 ( ± 0.3) 42.9 ( ± 0.5) BoRL is our Bayesian ➢ MAPOX 74.2 ( ± 0.4) 43.3 ( ± 0.4) optimization approach for learning rewards. BoRL 74.2 ( ± 0.2) 43.8 ( ± 0.2) MeRL achieves state-of-the-art ➢ MeRL 74.8 (± 0.2) 44.1 ( ± 0.2) results on WikiTableQuestions and WikiSQL , improving upon prior work by 1.2% and 2.4% respectively. Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

  18. Proprietary + Confidential Poster #49 tonight @Pacific Ballroom bit.ly/merl2019 Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Recommend

Outline Background Sparse Coding Semi-supervised Learning with Sparse Coding

Outline Background Sparse Coding Semi-supervised Learning with Sparse Coding

Sparse-coded Net Model and Applications Y. Gwon, M. Cha, W. Campbell, H.T. Kung, C. Dagli IEEE International Workshop on Machine Learning for Signal Processing (MLSP 2016) September 16, 2016 This work is sponsored by the

416 views • 17 slides
Good Machine Learning = Generalization Goal of machine learning: build models that generalize

Good Machine Learning = Generalization Goal of machine learning: build models that generalize

Good Machine Learning = Generalization Goal of machine learning: build models that generalize well to predicting new data Overfitting: fitting the training data too well, so we lose generality of model o Example: linear regression vs.

120 views • 11 slides
Monotonic Inference for Underspecified Episodic Logic Mandar Juvekar University of Rochester

Monotonic Inference for Underspecified Episodic Logic Mandar Juvekar University of Rochester

Monotonic Inference for Underspecified Episodic Logic Mandar Juvekar University of Rochester Natural Logic Meets Machine Learning 17 July 2020 Gene Louis Kim Lenhart K. Schubert UR UR Snchez Valencia Lambek Derivations Tableau-style

746 views • 38 slides
Learning Sparse Polynomials over product measures Kiran Vodrahalli knv2109@columbia.edu

Learning Sparse Polynomials over product measures Kiran Vodrahalli knv2109@columbia.edu

Learning Sparse Polynomials over product measures Kiran Vodrahalli knv2109@columbia.edu Columbia University December 11, 2017 The Problem Learning Sparse Polynomial Functions [Andoni, Panigrahy, Valiant, Zhang 14] Consider learning a

525 views • 15 slides
Taming the Beast: Topic imaging Predictive approach Sparse Machine Learning for Large Text

Taming the Beast: Topic imaging Predictive approach Sparse Machine Learning for Large Text

Sparse ML for Text 1/33 L. El Ghaoui Information Overload Taming the Beast: Topic imaging Predictive approach Sparse Machine Learning for Large Text Corpora Visualizaations Beyond co-occurence Examples Research Agenda Sparse PCA Laurent

488 views • 47 slides
Sparse optimization and machine learning in image reconstruction Aleksandra Pi zurica Group

Sparse optimization and machine learning in image reconstruction Aleksandra Pi zurica Group

Sparse optimization and machine learning in image reconstruction Aleksandra Pi zurica Group for Artificial Intelligence and Sparse Modelling (GAIM) Department Telecommunications and Information Processing Ghent University 2 nd Annual Meeting

821 views • 60 slides
Tutorial: Sparse Recovery Using Sparse Matrices Piotr Indyk MIT Problem Formulation

Tutorial: Sparse Recovery Using Sparse Matrices Piotr Indyk MIT Problem Formulation

Tutorial: Sparse Recovery Using Sparse Matrices Piotr Indyk MIT Problem Formulation (approximation theory, learning Fourier coeffs, linear sketching, finite rate of innovation, compressed sensing...) Setup: Data/signal in n-dimensional

418 views • 25 slides
Sparse dictionary learning in the presence of noise & outliers Rmi Gribonval INRIA Rennes

Sparse dictionary learning in the presence of noise & outliers Rmi Gribonval INRIA Rennes

Sparse dictionary learning in the presence of noise & outliers Rmi Gribonval INRIA Rennes - Bretagne Atlantique, France remi.gribonval@inria.fr Overview Context: sparse signal processing Dictionary learning Statistical

657 views • 37 slides
Sparse Matrices Beyond Solvers - Graphs, Biology, and Machine Learning (v2) Aydn Bulu

Sparse Matrices Beyond Solvers - Graphs, Biology, and Machine Learning (v2) Aydn Bulu

Sparse Matrices Beyond Solvers - Graphs, Biology, and Machine Learning (v2) Aydn Bulu Computational Research Division, LBNL EECS Department, UC Berkeley CS Summer Student Program July 16, 2020 Sparse Matrices I observed that most of

406 views • 39 slides
DSSTNE: Deep Learning At Scale For Large Sparse Datasets

DSSTNE: Deep Learning At Scale For Large Sparse Datasets

DSSTNE: Deep Learning At Scale For Large Sparse Datasets https://github.com/amznlabs/amazon-dsstne Scott Le Grand Senior Scientist Teza Technologies varelse2005@gmail.com Outline What's Deep Learning? Why GPUs? Deep Learning for

1.55k views • 76 slides
Bag of Pursuits and Neural Gas for Improved Sparse Coding Manifold Learning with Sparse Coding

Bag of Pursuits and Neural Gas for Improved Sparse Coding Manifold Learning with Sparse Coding

Bag of Pursuits and Neural Gas for Improved Sparse Coding Manifold Learning with Sparse Coding Thomas Martinetz Institute for Neuro- and Bioinformatics University of L ubeck 26.8.2010 1 / 27 Thomas Martinetz Bag of Pursuits and Neural

582 views • 27 slides
Learning ancestral atom of structured dictionary via sparse coding Bernoulli Society Satellite

Learning ancestral atom of structured dictionary via sparse coding Bernoulli Society Satellite

Learning ancestral atom of structured dictionary via sparse coding Bernoulli Society Satellite Meeting Noboru Murata Waseda University 2 September, 2013 joint work with Toshimitsu Aritake, Hideitsu Hino 1 / 25 sparse coding a methodology

513 views • 25 slides
Sparse Coding and Dictionary Learning for Image Analysis Part IV: New sparse models Francis

Sparse Coding and Dictionary Learning for Image Analysis Part IV: New sparse models Francis

Sparse Coding and Dictionary Learning for Image Analysis Part IV: New sparse models Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro ICCV09 tutorial, Kyoto, 28th September 2009 Francis Bach, Julien Mairal, Jean Ponce and

353 views • 19 slides
Sparse Matrices Example Of Sparse Matrices diagonal tridiagonal sparse many elements are

Sparse Matrices Example Of Sparse Matrices diagonal tridiagonal sparse many elements are

Sparse Matrices Example Of Sparse Matrices diagonal tridiagonal sparse many elements are zero lower triangular (?) dense few elements are zero These are structured sparse matrices. May be mapped into a 1D array so that a mapping

391 views • 6 slides
Image Reconstruction Tutorial Part 1: Sparse optimization and learning approaches zurica 1 and

Image Reconstruction Tutorial Part 1: Sparse optimization and learning approaches zurica 1 and

Image Reconstruction Tutorial Part 1: Sparse optimization and learning approaches zurica 1 and Bart Goossens 2 Aleksandra Pi 1 Group for Artificial Intelligence and Sparse Modelling (GAIM) 2 Image Processing and Interpretation (IPI) group

837 views • 54 slides
Sparse Coding and Dictionary Learning for Image Analysis Part I: Optimization for Sparse Coding

Sparse Coding and Dictionary Learning for Image Analysis Part I: Optimization for Sparse Coding

Sparse Coding and Dictionary Learning for Image Analysis Part I: Optimization for Sparse Coding Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro ICCV09 tutorial, Kyoto, 28th September 2009 Francis Bach, Julien Mairal, Jean Ponce

621 views • 41 slides
Sparse Matrices sparse many elements are zero dense few elements are zero Example Of

Sparse Matrices sparse many elements are zero dense few elements are zero Example Of

Sparse Matrices sparse many elements are zero dense few elements are zero Example Of Sparse Matrices diagonal tridiagonal lower triangular (?) These are structured sparse matrices. May be mapped into a 1D array so that a mapping

698 views • 11 slides
Applying Deep Learning To Airbnb Search Unique Challenges High Value Transactions Sparse

Applying Deep Learning To Airbnb Search Unique Challenges High Value Transactions Sparse

MALAY HALDAR / April 2019 / QCon Applying Deep Learning To Airbnb Search Unique Challenges High Value Transactions Sparse Listing Data Sparse User Data Users are picky when spending Each listing can be booked at Travel is not a daily use

622 views • 31 slides
Sparse Fuzzy Techniques There Is Room for . . . Our Idea Improve Machine Learning Towards an

Sparse Fuzzy Techniques There Is Room for . . . Our Idea Improve Machine Learning Towards an

Machine Learning: A . . . Machine Learning: A . . . Machine Learning: . . . Sparse Fuzzy Techniques There Is Room for . . . Our Idea Improve Machine Learning Towards an Efficient . . . Taking the Specific . . . Reinaldo Sanchez 1 , Christian

577 views • 16 slides
Machine Learning and Sparsity Klaus-Robert Mller !!et al.!! Todays Talk sensing, sparse

Machine Learning and Sparsity Klaus-Robert Mller !!et al.!! Todays Talk sensing, sparse

Machine Learning and Sparsity Klaus-Robert Mller !!et al.!! Todays Talk sensing, sparse models and generalization interpretabilty and sparse methods explaining for nonlinear methods Sparse Models & Generalization? Machine

458 views • 25 slides
Compressed Sensing Meets Machine Learning - Classification of Mixture Subspace Models via Sparse

Compressed Sensing Meets Machine Learning - Classification of Mixture Subspace Models via Sparse

Introduction Classification via Sparse Representation Distributed Pattern Recognition Conclusion Compressed Sensing Meets Machine Learning - Classification of Mixture Subspace Models via Sparse Representation Allen Y. Yang

861 views • 31 slides
On the Use of Underspecified Data-Type Semantics for Type Safety in Low-Level Code Hendrik Tews 1

On the Use of Underspecified Data-Type Semantics for Type Safety in Low-Level Code Hendrik Tews 1

Introduction Basics Type Errors Stronger Encodings Type Sensitivity Conclusion On the Use of Underspecified Data-Type Semantics for Type Safety in Low-Level Code Hendrik Tews 1 , Marcus V olp 1 , Tjark Weber 2 1 Technische Universit at

559 views • 40 slides
Dictionary Learning for Graph Signals 236862 Introduction to Sparse and Redundant

Dictionary Learning for Graph Signals 236862 Introduction to Sparse and Redundant

Dictionary Learning for Graph Signals 236862 Introduction to Sparse and Redundant Representations joint work with Yael Yankelevsky 22.12.2019 Prof. Michael Elad The Sparseland Model Dictionary Learning: = Model assumption: All data

764 views • 35 slides
Learning Effective and Interpretable Semantic Models using Non-Negative Sparse Embedding (NNSE)

Learning Effective and Interpretable Semantic Models using Non-Negative Sparse Embedding (NNSE)

1 Learning Effective and Interpretable Semantic Models using Non-Negative Sparse Embedding (NNSE) Brian Murphy, Partha Talukdar, Tom Mitchell Machine Learning Department Carnegie Mellon University 2 Distributional Semantic Modeling 2

1.03k views • 86 slides

More recommend