cheap tricks and the perils of machine learning
play

Cheap Tricks and the Perils of Machine Learning Percy Liang - PowerPoint PPT Presentation

Oct 05, 2023 •268 likes •1.16k views

Cheap Tricks and the Perils of Machine Learning Percy Liang Stanford / (Semantic Machines / Microsoft) NAACL Workshop on New Forms of Generalization June 5, 2018 [with Pranav Rajpurkar et al; 2016] Reading Comprehension (SQuAD) 1 2 [with

  1. Cheap Tricks and the Perils of Machine Learning Percy Liang Stanford / (Semantic Machines / Microsoft) NAACL Workshop on New Forms of Generalization — June 5, 2018

  2. [with Pranav Rajpurkar et al; 2016] Reading Comprehension (SQuAD) 1

  3. 2

  4. [with Robin Jia; EMNLP 2017] Reading comprehension The way a teacher promotes the course they are teaching, the more the student will get out of the subject matter. The three most important aspects of teacher enthusiasm are enthusiasm about teaching, enthusiasm about the students, and enthusiasm about the subject matter. A teacher must enjoy teaching. If they do not enjoy what they are doing, the students will be able to tell. They also must enjoy being around their students. A teacher who cares for their students is going to help that individual succeed in their life in the future. The teacher also needs to be enthusiastic about the subject matter they are teaching. For example, a teacher talking about chemistry needs to enjoy the art of chemistry and show that to their students. A spark in the teacher may create a spark of excitement in the student as well. An enthusiastic teacher has the ability to be very influential in the young student ’s life. What can an enthusiastic teacher be to a young student? r-net+ [MSR-A] very influential 3

  5. [with Robin Jia; EMNLP 2017] Reading comprehension The way a teacher promotes the course they are teaching, the more the student will get out of the subject matter. The three most important aspects of teacher enthusiasm are enthusiasm about teaching, enthusiasm about the students, and enthusiasm about the subject matter. A teacher must enjoy teaching. If they do not enjoy what they are doing, the students will be able to tell. They also must enjoy being around their students. A teacher who cares for their students is going to help that individual succeed in their life in the future. The teacher also needs to be enthusiastic about the subject matter they are teaching. For example, a teacher talking about chemistry needs to enjoy the art of chemistry and show that to their students. A spark in the teacher may create a spark of excitement in the student as well. An enthusiastic teacher has the ability to be very influential in the young student ’s life. An unenthusiastic teacher can be troubling to a young student. What can an enthusiastic teacher be to a young student? r-net+ [MSR-A] 3

  6. [with Robin Jia; EMNLP 2017] Reading comprehension The way a teacher promotes the course they are teaching, the more the student will get out of the subject matter. The three most important aspects of teacher enthusiasm are enthusiasm about teaching, enthusiasm about the students, and enthusiasm about the subject matter. A teacher must enjoy teaching. If they do not enjoy what they are doing, the students will be able to tell. They also must enjoy being around their students. A teacher who cares for their students is going to help that individual succeed in their life in the future. The teacher also needs to be enthusiastic about the subject matter they are teaching. For example, a teacher talking about chemistry needs to enjoy the art of chemistry and show that to their students. A spark in the teacher may create a spark of excitement in the student as well. An enthusiastic teacher has the ability to be very influential in the young student ’s life. An unenthusiastic teacher can be troubling to a young student. What can an enthusiastic teacher be to a young student? r-net+ [MSR-A] troubling 3

  7. Humans versus machines 4

  8. Harder tests? We want multiple-choice questions that people can answer easily. But we also want to avoid as much as possible questions that can be answered using cheap tricks (aka heuristics). — Hector Levesque, 2013 5

  9. Winograd schema The dog chased the cat , which ran up a tree. It waited at the top. 6

  10. Winograd schema The dog chased the cat , which ran up a tree. It waited at the top. The dog chased the cat , which ran up a tree. It waited at the bottom. 6

  11. [hat tip: Fernando Pereira] Winograd schema The dog chased the cat , which ran up a tree. It waited at the top. 7

  12. [hat tip: Fernando Pereira] Winograd schema The dog chased the cat , which ran up a tree. It waited at the top. 7

  13. [hat tip: Fernando Pereira] Winograd schema The dog chased the cat , which ran up a tree. It waited at the top. 7

  14. [hat tip: Fernando Pereira] Winograd schema The dog chased the cat , which ran up a tree. It waited at the top. 7

  15. [hat tip: Fernando Pereira] Winograd schema The dog chased the cat , which ran up a tree. It waited at the bottom. 8

  16. [hat tip: Fernando Pereira] Winograd schema The dog chased the cat , which ran up a tree. It waited at the bottom. 8

  17. [hat tip: Fernando Pereira] Winograd schema The dog chased the cat , which ran up a tree. It waited at the bottom. 8

  18. [hat tip: Fernando Pereira] Winograd schema The dog chased the cat , which ran up a tree. It waited at the bottom. 8

  19. But what about... Commonsense knowledge Logical reasoning Linguistic phenomena Intuitive physics ... 9

  20. But what about... Commonsense knowledge Logical reasoning Linguistic phenomena Intuitive physics ... MACHINE LEARNING DOESN’T CARE 9

  21. Interpolation is insufficient ≈ Train Test 10

  22. Interpolation is insufficient ≈ Train Test Any expressive model with enough data will do the job 10

  23. Extrapolation is harder �≈ Train Test 11

  24. Extrapolation is harder �≈ Train Test 11

  25. Extrapolation is harder �≈ Train Test To extrapolate (and be robust), must get a more ”correct” model 11

  26. Outline Harder data Stronger models 12

  27. Adversarial evaluation of reading comprehension (EMNLP 2017) Robin Jia 13

  28. [with Robin Jia; EMNLP 2017] Reading comprehension The way a teacher promotes the course they are teaching, the more the student will get out of the subject matter. The three most important aspects of teacher enthusiasm are enthusiasm about teaching, enthusiasm about the students, and enthusiasm about the subject matter. A teacher must enjoy teaching. If they do not enjoy what they are doing, the students will be able to tell. They also must enjoy being around their students. A teacher who cares for their students is going to help that individual succeed in their life in the future. The teacher also needs to be enthusiastic about the subject matter they are teaching. For example, a teacher talking about chemistry needs to enjoy the art of chemistry and show that to their students. A spark in the teacher may create a spark of excitement in the student as well. An enthusiastic teacher has the ability to be very influential in the young student ’s life. What can an enthusiastic teacher be to a young student? r-net+ [MSR-A] very influential 14

  29. [with Robin Jia; EMNLP 2017] Reading comprehension The way a teacher promotes the course they are teaching, the more the student will get out of the subject matter. The three most important aspects of teacher enthusiasm are enthusiasm about teaching, enthusiasm about the students, and enthusiasm about the subject matter. A teacher must enjoy teaching. If they do not enjoy what they are doing, the students will be able to tell. They also must enjoy being around their students. A teacher who cares for their students is going to help that individual succeed in their life in the future. The teacher also needs to be enthusiastic about the subject matter they are teaching. For example, a teacher talking about chemistry needs to enjoy the art of chemistry and show that to their students. A spark in the teacher may create a spark of excitement in the student as well. An enthusiastic teacher has the ability to be very influential in the young student ’s life. An unenthusiastic teacher can be troubling to a young student. What can an enthusiastic teacher be to a young student? r-net+ [MSR-A] 14

  30. [with Robin Jia; EMNLP 2017] Reading comprehension The way a teacher promotes the course they are teaching, the more the student will get out of the subject matter. The three most important aspects of teacher enthusiasm are enthusiasm about teaching, enthusiasm about the students, and enthusiasm about the subject matter. A teacher must enjoy teaching. If they do not enjoy what they are doing, the students will be able to tell. They also must enjoy being around their students. A teacher who cares for their students is going to help that individual succeed in their life in the future. The teacher also needs to be enthusiastic about the subject matter they are teaching. For example, a teacher talking about chemistry needs to enjoy the art of chemistry and show that to their students. A spark in the teacher may create a spark of excitement in the student as well. An enthusiastic teacher has the ability to be very influential in the young student ’s life. An unenthusiastic teacher can be troubling to a young student. What can an enthusiastic teacher be to a young student? r-net+ [MSR-A] troubling 14

  31. worksheets.codalab.org submit model to evaluate on hidden test set 15

  32. Results on SQuAD models Model Original F1 Adversarial F1 SLQA+ 88.6 64.2 r-net+ 88.5 63.4 ReasoNet-E 81.1 49.8 SEDT-E 80.1 46.5 BiDAF-E 80.0 46.9 Mnemonic-E 79.1 55.3 Ruminating 78.8 47.7 jNet 78.6 47.0 Mnemonic-S 78.5 56.0 ReasoNet-S 78.2 50.3 MPCM-S 77.0 50.0 16

Recommend

Beating the Perils of Non-Convexity: Machine Learning using Tensor Methods Anima Anandkumar ..

Beating the Perils of Non-Convexity: Machine Learning using Tensor Methods Anima Anandkumar ..

Beating the Perils of Non-Convexity: Machine Learning using Tensor Methods Anima Anandkumar .. Joint work with Majid Janzamin and Hanie Sedghi. U.C. Irvine Learning with Big Data Learning is finding needle in a haystack Learning with Big

726 views • 60 slides
Machine Learning Tricks Philipp Koehn 13 October 2020 Philipp Koehn Machine Translation:

Machine Learning Tricks Philipp Koehn 13 October 2020 Philipp Koehn Machine Translation:

Machine Learning Tricks Philipp Koehn 13 October 2020 Philipp Koehn Machine Translation: Machine Learning Tricks 13 October 2020 Machine Learning 1 Myth of machine learning given: real world examples automatically build model

928 views • 53 slides
Adversarial Machine Learning Daniel Lowd University of Oregon

Adversarial Machine Learning Daniel Lowd University of Oregon

Adversarial Machine Learning Daniel Lowd University of Oregon Example: Spam Filtering From: spammer@example.com 1. Cheap mortgage now!!! Feature Weights cheap = 1.0

316 views • 14 slides
An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning http://www.cs.iastate.edu/~cs573x/bbsilab.html Machine Learning Software Preparing Data Building Classifiers Interpreting Results Machine Learning Software

496 views • 26 slides
Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by

Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by

Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by which computers can be trained through observation, rather than being explicitly programmed. Basically, a program is given input and adjusts its

559 views • 17 slides
Machine Learning: Study of algorithms that improve their performance P at some task T

Machine Learning: Study of algorithms that improve their performance P at some task T

Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 12, 2015 Today: Readings: What is machine learning? The Discipline of ML Decision tree learning Mitchell, Chapter 3

872 views • 29 slides
Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning

Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning

GCT634/AI613: Musical Applications of Machine Learning (Fall 2020) Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning Pipeline in Classification Tasks A set of hand-designed audio features are selected

401 views • 26 slides
CS 335 Machine Learning What is Machine Learning? Dan Sheldon Spring 2019 What is Machine

CS 335 Machine Learning What is Machine Learning? Dan Sheldon Spring 2019 What is Machine

1/22/19 CS 335 Machine Learning What is Machine Learning? Dan Sheldon Spring 2019 What is Machine Learning? A Simple Task: Recognize Obama How do you program a computer to Recognize faces? Recommend movies? Decide which web

582 views • 5 slides
Businesses and Tax The Perils of Perception October 2015 Business and Tax Perils of

Businesses and Tax The Perils of Perception October 2015 Business and Tax Perils of

Businesses and Tax The Perils of Perception October 2015 Business and Tax Perils of Perception | October 2015 | FINAL | Public 1 Business contributes the largest proportion of tax of any group of tax payers (29%), yet people massively

270 views • 14 slides
Machine Learning Machine Learning: algorithms that use experience to improve their

Machine Learning Machine Learning: algorithms that use experience to improve their

Machine Learning Machine Learning: algorithms that use experience to improve their performance We use machine learning in situations where it is very challenging (or impossible) to define the rules by hand: e.g. face detection

1.04k views • 28 slides
Social Learning in edX ABSTRACT edX was born following two revolutions in technology: cheap,

Social Learning in edX ABSTRACT edX was born following two revolutions in technology: cheap,

Social Learning in edX ABSTRACT edX was born following two revolutions in technology: cheap, robust video streaming and the growth of on-line social, in a belief that this progress would allow us to create effective, at-scale digital learning

778 views • 45 slides
Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Introduction to Machine Learning 1 / 12 Outline 1 Nearest Neighbor Prediction 2 Complexity Considerations 3 The Voronoi Diagram 4 Overfitting

565 views • 12 slides
1 Why Study Machine Learning? Why Study Machine Learning? Cognitive Science The Time is Ripe

1 Why Study Machine Learning? Why Study Machine Learning? Cognitive Science The Time is Ripe

What is Learning? Herbert Simon: Learning is any process by which a system improves performance from CS 391L: Machine Learning experience. Introduction What is the task? Classification Problem solving / planning /

603 views • 6 slides
MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING

MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING

JACOPO DI IORIO, MATTEO FONTANA, DANIELE OCCHIUTO, CLAUDIA VOLPETTI MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING STATISTICAL LEARNING Separate evolutions, but with shared methodologies MERGE

565 views • 55 slides
Apache PredictionIO End-to-End Machine Learning Server with Apache Spark What is Machine

Apache PredictionIO End-to-End Machine Learning Server with Apache Spark What is Machine

Apache PredictionIO End-to-End Machine Learning Server with Apache Spark What is Machine Learning? What is Machine Learning? Examples of machine learning applications: Facial recognition software E-mail spam detectors Automated

302 views • 17 slides
Machine Learning 11 AI Slides (6e) c Lin Zuoquan@PKU 1998-2020 11 1 11 Machine Learning

Machine Learning 11 AI Slides (6e) c Lin Zuoquan@PKU 1998-2020 11 1 11 Machine Learning

Machine Learning 11 AI Slides (6e) c Lin Zuoquan@PKU 1998-2020 11 1 11 Machine Learning 11.1 Learning agents 11.2 Inductive learning 11.3 Deep learning 11.4 Statistical learning 11.5 Reinforcement learning 11.6 Transfer learning

1.7k views • 159 slides
Deep Learning: Intro Juhan Nam Review of Traditional Machine Learning The traditional machine

Deep Learning: Intro Juhan Nam Review of Traditional Machine Learning The traditional machine

GCT634/AI613: Musical Applications of Machine Learning (Fall 2020) Deep Learning: Intro Juhan Nam Review of Traditional Machine Learning The traditional machine learning pipeline Temporal Frame-level Unsupervised Classifier Summary

613 views • 27 slides
Machine Learning for Auto Optimization What is Machine Learning? Definition: Machine

Machine Learning for Auto Optimization What is Machine Learning? Definition: Machine

Machine Learning for Auto Optimization What is Machine Learning? Definition: Machine learning refers to any system where the performance of a machine in performing a task improves by gaining more experience in performing that task .

394 views • 27 slides
MACHINE LEARNING kernels 1 MACHINE LEARNING 2012 MACHINE LEARNING Kernels: Intuition How

MACHINE LEARNING kernels 1 MACHINE LEARNING 2012 MACHINE LEARNING Kernels: Intuition How

MACHINE LEARNING 2012 MACHINE LEARNING MACHINE LEARNING kernels 1 MACHINE LEARNING 2012 MACHINE LEARNING Kernels: Intuition How to separate the red class from the grey class? x 2 360 r x 1 Polar coordinates Data

1.04k views • 44 slides
BBM406 Fundamentals of Machine Learning Lecture 2: Machine Learning by Examples, Nearest

BBM406 Fundamentals of Machine Learning Lecture 2: Machine Learning by Examples, Nearest

photo:@rewardyfahmi // Unsplash BBM406 Fundamentals of Machine Learning Lecture 2: Machine Learning by Examples, Nearest Neighbor Classifier Aykut Erdem // Hacettepe University // Fall 2019 When Do We Use Machine Learning? ML is

1.33k views • 110 slides
Machine Learning Modeling and Learning 15-110 Monday 4/13 Learning Goals Given a

Machine Learning Modeling and Learning 15-110 Monday 4/13 Learning Goals Given a

Machine Learning Modeling and Learning 15-110 Monday 4/13 Learning Goals Given a dataset, identify features which may help predict information about the data Identify common types of machine learning algorithms 2 Machine

553 views • 27 slides
Machine Learning @ Amazon Ralf Herbrich Amazon 6/29/17 1 Overview Machine Learning in

Machine Learning @ Amazon Ralf Herbrich Amazon 6/29/17 1 Overview Machine Learning in

Machine Learning @ Amazon Ralf Herbrich Amazon 6/29/17 1 Overview Machine Learning in Practise Probabilities Finite Resource Machine Learning @ Amazon Forecasting Machine Translation Visual Systems Conclusions

348 views • 34 slides
Softmax Classifier + SGD Todays Class Intro to Machine Learning What is Machine Learning?

Softmax Classifier + SGD Todays Class Intro to Machine Learning What is Machine Learning?

CS6501: Deep Learning for Visual Recognition Softmax Classifier + SGD Todays Class Intro to Machine Learning What is Machine Learning? Supervised Learning: Classification with K-nearest neighbors Unsupervised Learning: Clustering with

1.36k views • 91 slides
Neural Networks for Machine Learning Lecture 1a Why do we need machine learning? Geoffrey Hinton

Neural Networks for Machine Learning Lecture 1a Why do we need machine learning? Geoffrey Hinton

Neural Networks for Machine Learning Lecture 1a Why do we need machine learning? Geoffrey Hinton with Nitish Srivastava Kevin Swersky What is Machine Learning? It is very hard to write programs that solve problems like recognizing a

514 views • 48 slides

More recommend