learning from data lecture 14 three learning principles
play

Learning From Data Lecture 14 Three Learning Principles Occams - PowerPoint PPT Presentation

Aug 17, 2023 •420 likes •1.01k views

Learning From Data Lecture 14 Three Learning Principles Occams Razor Sampling Bias Data Snooping M. Magdon-Ismail CSCI 4100/6100 recap: Validation and Cross Validation Validation Cross Validation D ( N ) D 1 D 2 D N D train D

  1. Learning From Data Lecture 14 Three Learning Principles Occam’s Razor Sampling Bias Data Snooping M. Magdon-Ismail CSCI 4100/6100

  2. recap: Validation and Cross Validation Validation Cross Validation D ( N ) D 1 D 2 · · · D N D train D ( N − K ) · · · g 1 g 2 g N ( x 1 , y 1 ) ( x 2 , y 2 ) ( x N , y N ) g D val · · · ( K ) e 1 e 2 e N � �� � take average g E val ( g ) g E cv · · · H 1 H 2 H 3 H M Model Selection − − − − − − − − − − − − → → → → · · · g 1 g 2 g 3 g M M Three Learning Principles : 2 /58 � A c L Creator: Malik Magdon-Ismail Occam, bias, snooping − →

  3. We Will Discuss . . . • Occam’s Razor : pick a model carefully • Sampling Bias : generate the data carefuly • Data Snooping : handle the data carefully M Three Learning Principles : 3 /58 � A c L Creator: Malik Magdon-Ismail Occam’s Razor − →

  4. Occam’s Razor M Three Learning Principles : 4 /58 � A c L Creator: Malik Magdon-Ismail Occam − →

  5. Occam’s Razor use a ‘ razor ’ to ‘trim down’ “an explanation of the data to make it as simple as possible but no simpler.” attributed to William of Occam (14th Century) and often mistakenly to Einstein M Three Learning Principles : 5 /58 � A c L Creator: Malik Magdon-Ismail Simpler is Better − →

  6. Simpler is Better The simplest model that fits the data is also the most plausible. . . . or, beware of using complex models to fit data M Three Learning Principles : 6 /58 � A c L Creator: Malik Magdon-Ismail What is Simpler? − →

  7. What is Simpler? simple hypothesis h simple hypothesis set H Ω( h ) Ω( H ) low order polynomial H with small d vc hypothesis with small weights small number of hypotheses easily described hypothesis low entropy set . . . . . . The equivalence: A hypothesis set with simple hypotheses must be small We had a glimpse of this: λ soft order constraint (smaller H ) → minimize E aug (favors simpler h ) . ← − − − − M Three Learning Principles : 7 /58 � A c L Creator: Malik Magdon-Ismail What is Simpler? − →

  8. What is Simpler? simple hypothesis h simple hypothesis set H Ω( h ) Ω( H ) low order polynomial H with small d vc hypothesis with small weights small number of hypotheses easily described hypothesis low entropy set . . . . . . The equivalence: A hypothesis set with simple hypotheses must be small We had a glimpse of this: λ soft order constraint (smaller H ) → minimize E aug (favors simpler h ) . ← − − − − M Three Learning Principles : 8 /58 � A c L Creator: Malik Magdon-Ismail What is Simpler? − →

  9. What is Simpler? simple hypothesis h simple hypothesis set H Ω( h ) Ω( H ) low order polynomial H with small d vc hypothesis with small weights small number of hypotheses easily described hypothesis low entropy set . . . . . . The equivalence: A hypothesis set with simple hypotheses must be small We had a glimpse of this: λ soft order constraint (smaller H ) ← − − − − → minimize E aug (favors simpler h ) . M Three Learning Principles : 9 /58 � A c L Creator: Malik Magdon-Ismail Why is Simpler Better − →

  10. Why is Simpler Better Mathematically: simple curtails ability to fit noise, VC-dimension is small, and blah and blah . . . simpler is better because you will be more “surprised” when you fit the data. If something unlikely happens, it is very significant when it happens. . . . “Is there any other point to which you would wish to draw my attention?” Detective Gregory: Sherlock Holmes: “To the curious incident of the dog in the night-time.” “The dog did nothing in the night-time.” Detective Gregory: “That was the curious incident.” Sherlock Holmes: . . . – Silver Blaze , Sir Arthur Conan Doyle M Three Learning Principles : 10 /58 � A c L Creator: Malik Magdon-Ismail Scientific Experiment − →

  11. A Scientific Experiment Axiom. If an experiment has no chance of falsifying a hypothesis, then the result of that experiment provides no evidence one way or the other for the hypothesis. Scientist 3 resistivity ρ temperature T no evidence very convincing some evidence? Who provides most evidence for the hypothesis “ ρ is linear in T ”? M Three Learning Principles : 11 /58 � A c L Creator: Malik Magdon-Ismail Scientific Experiment − →

  12. A Scientific Experiment Axiom. If an experiment has no chance of falsifying a hypothesis, then the result of that experiment provides no evidence one way or the other for the hypothesis. Scientist 2 Scientist 3 resistivity ρ resistivity ρ temperature T temperature T no evidence very convincing some evidence? Who provides most evidence for the hypothesis “ ρ is linear in T ”? M Three Learning Principles : 12 /58 � A c L Creator: Malik Magdon-Ismail Scientific Experiment − →

  13. A Scientific Experiment Axiom. If an experiment has no chance of falsifying a hypothesis, then the result of that experiment provides no evidence one way or the other for the hypothesis. Scientist 1 Scientist 2 Scientist 3 resistivity ρ resistivity ρ resistivity ρ temperature T temperature T temperature T no evidence very convincing some evidence? Who provides most evidence for the hypothesis “ ρ is linear in T ”? M Three Learning Principles : 13 /58 � A c L Creator: Malik Magdon-Ismail Scientific Experiment − →

  14. A Scientific Experiment Axiom. If an experiment has no chance of falsifying a hypothesis, then the result of that experiment provides no evidence one way or the other for the hypothesis. Scientist 1 Scientist 2 Scientist 3 resistivity ρ resistivity ρ resistivity ρ temperature T temperature T temperature T no evidence very convincing some evidence? Who provides most evidence for the hypothesis “ ρ is linear in T ”? M Three Learning Principles : 14 /58 � A c L Creator: Malik Magdon-Ismail Scientist 2 vs. 3 − →

  15. Scientist 2 Versus Scientist 3 Axiom. If an experiment has no chance of falsifying a hypothesis, then the result of that experiment provides no evidence one way or the other for the hypothesis. Scientist 1 Scientist 2 Scientist 3 resistivity ρ resistivity ρ resistivity ρ temperature T temperature T temperature T no evidence very convincing some evidence? Who provides most evidence? M Three Learning Principles : 15 /58 � A c L Creator: Malik Magdon-Ismail Scientist 1 vs. 3 − →

  16. Scientist 1 versus Scientist 3 Axiom. If an experiment has no chance of falsifying a hypothesis, then the result of that experiment provides no evidence one way or the other for the hypothesis. Scientist 1 Scientist 2 Scientist 3 resistivity ρ resistivity ρ resistivity ρ temperature T temperature T temperature T no evidence very convincing some evidence? Who provides most evidence? M Three Learning Principles : 16 /58 � A c L Creator: Malik Magdon-Ismail Non-Falsifiability − →

  17. Axiom of Non-Falsifiability Axiom. If an experiment has no chance of falsifying a hypothesis, then the result of that experiment provides no evidence one way or the other for the hypothesis. Scientist 1 Scientist 2 Scientist 3 resistivity ρ resistivity ρ temperature T temperature T no evidence very convincing some evidence? Who provides most evidence? M Three Learning Principles : 17 /58 � A c L Creator: Malik Magdon-Ismail Falsification and m H ( N ) − →

  18. Falsification and m H ( N ) If H shatters x 1 , · · · , x N , – Don’t be surprised if you fit the data. – Can’t falsify “ H is a good set of candidate hypotheses for f ”. If H doesn’t shatter x 1 , · · · , x N , and the target values are uniformly distributed, P [falsification] ≥ 1 − m H ( N ) . 2 N A good fit is surprising with simple H , hence significant. You can, but didn’t falsify “ H is a good set of candidate hypotheses for f ” The data must have a chance to win. M Three Learning Principles : 18 /58 � A c L Creator: Malik Magdon-Ismail Falsification and m H ( N ) − →

  19. Falsification and m H ( N ) If H shatters x 1 , · · · , x N , – Don’t be surprised if you fit the data. – Can’t falsify “ H is a good set of candidate hypotheses for f ”. If H doesn’t shatter x 1 , · · · , x N , and the target values are uniformly distributed, P [falsification] ≥ 1 − m H ( N ) . 2 N A good fit is surprising with simple H , hence significant. You can, but didn’t falsify “ H is a good set of candidate hypotheses for f ” The data must have a chance to win. M Three Learning Principles : 19 /58 � A c L Creator: Malik Magdon-Ismail Falsification and m H ( N ) − →

  20. Falsification and m H ( N ) If H shatters x 1 , · · · , x N , – Don’t be surprised if you fit the data. – Can’t falsify “ H is a good set of candidate hypotheses for f ”. If H doesn’t shatter x 1 , · · · , x N , and the target values are uniformly distributed, P [falsification] ≥ 1 − m H ( N ) . 2 N A good fit is surprising with simple H , hence significant. You can, but didn’t falsify “ H is a good set of candidate hypotheses for f ” The data must have a chance to win. M Three Learning Principles : 20 /58 � A c L Creator: Malik Magdon-Ismail Beyond Occam − →

Recommend

Learning From Data Lecture 2 The Perceptron The Learning Setup A Simple Learning Algorithm: PLA

Learning From Data Lecture 2 The Perceptron The Learning Setup A Simple Learning Algorithm: PLA

Learning From Data Lecture 2 The Perceptron The Learning Setup A Simple Learning Algorithm: PLA Other Views of Learning Is Learning Feasible: A Puzzle M. Magdon-Ismail CSCI 4100/6100 recap: The Plan 1. What is Learning? 2. Can We do it?

613 views • 25 slides
Learning From Data Lecture 3 Is Learning Feasible? Outside the Data Probability to the Rescue

Learning From Data Lecture 3 Is Learning Feasible? Outside the Data Probability to the Rescue

Learning From Data Lecture 3 Is Learning Feasible? Outside the Data Probability to the Rescue Learning vs. Verification Selection Bias - A Cartoon M. Magdon-Ismail CSCI 4100/6100 recap: The Perceptron Learning Algorithm y = +1 y x

538 views • 27 slides
Learning From Data Lecture 15 Reflecting on Our Path - Epilogue to Part I What We Did The

Learning From Data Lecture 15 Reflecting on Our Path - Epilogue to Part I What We Did The

Learning From Data Lecture 15 Reflecting on Our Path - Epilogue to Part I What We Did The Machine Learning Zoo Moving Forward M. Magdon-Ismail CSCI 4100/6100 recap: Three Learning Principles Scientist 1 Scientist 2 Scientist 3 resistivity

392 views • 11 slides
CSE217 INTRODUCTION TO DATA SCIENCE LECTURE 6: LEARNING PRINCIPLES Spring 2019 Marion Neumann

CSE217 INTRODUCTION TO DATA SCIENCE LECTURE 6: LEARNING PRINCIPLES Spring 2019 Marion Neumann

CSE217 INTRODUCTION TO DATA SCIENCE LECTURE 6: LEARNING PRINCIPLES Spring 2019 Marion Neumann RECAP: MACHINE LEARNING Workflow 2 NOISE noisy samples from true function 3 WHY IS NOISE A PROBLEM? small random sample from the noisy

746 views • 19 slides
Occams Razor Sampling Bias : generate the data carefuly Data Snooping : handle the data

Occams Razor Sampling Bias : generate the data carefuly Data Snooping : handle the data

recap: Validation and Cross Validation Validation Cross Validation D ( N ) D 1 D 2 D N D train Learning From Data D ( N K ) g 1 g 2 g N Lecture 14 ( x 1 , y 1 ) ( x 2 , y 2 ) ( x N , y N ) Three Learning Principles D

486 views • 15 slides
Learning Aides Additional tools that can be applied to all techniques Learning From Data Lecture

Learning Aides Additional tools that can be applied to all techniques Learning From Data Lecture

Learning Aides Additional tools that can be applied to all techniques Learning From Data Lecture 27 Learning Aides Input Preprocessing Dimensionality Reduction and Feature Selection Preprocess data to account for arbitrary choices during data

430 views • 4 slides
Learning From Data Lecture 27 Learning Aides Input Preprocessing Dimensionality Reduction and

Learning From Data Lecture 27 Learning Aides Input Preprocessing Dimensionality Reduction and

Learning From Data Lecture 27 Learning Aides Input Preprocessing Dimensionality Reduction and Feature Selection Principal Components Analysis (PCA) Hints, Data Cleaning, Validation, . . . M. Magdon-Ismail CSCI 4100/6100 Learning Aides

271 views • 16 slides
Learning From Data Lecture 25 The Kernel Trick Learning with only inner products The Kernel M.

Learning From Data Lecture 25 The Kernel Trick Learning with only inner products The Kernel M.

Learning From Data Lecture 25 The Kernel Trick Learning with only inner products The Kernel M. Magdon-Ismail CSCI 4100/6100 recap: Large Margin is Better Controling Overfitting Non-Separable Data 0.08 random hyperplane 2 w t w + C N E

576 views • 18 slides
Learning From Data Lecture 19 A Peek At Unsupervised Learning k -Means Clustering Probability

Learning From Data Lecture 19 A Peek At Unsupervised Learning k -Means Clustering Probability

Learning From Data Lecture 19 A Peek At Unsupervised Learning k -Means Clustering Probability Density Estimation Gaussian Mixture Models M. Magdon-Ismail CSCI 4100/6100 recap: Radial Basis Functions Nonparametric RBF Parametric k -RBF-Network

253 views • 23 slides
Learning From Data Lecture 1 The Learning Problem Introduction Motivation Credit Default - A

Learning From Data Lecture 1 The Learning Problem Introduction Motivation Credit Default - A

Learning From Data Lecture 1 The Learning Problem Introduction Motivation Credit Default - A Running Example Summary of the Learning Problem M. Magdon-Ismail CSCI 4100/6100 Resources 1. Web Page: www.cs.rpi.edu/ magdon/courses/learn.php

499 views • 16 slides
Learning of concepts and principles of IT Aim Determine the level of mastery of the

Learning of concepts and principles of IT Aim Determine the level of mastery of the

INF3280 7 February 2014 Learning of concepts and principles of IT Aim Determine the level of mastery of the learning goals for Assignment 2 according to the learning model Core literature: Chapter 5. Learning IT

260 views • 12 slides
Learning From Data Lecture 5 Training Versus Testing The Two Questions of Learning Theory of

Learning From Data Lecture 5 Training Versus Testing The Two Questions of Learning Theory of

Learning From Data Lecture 5 Training Versus Testing The Two Questions of Learning Theory of Generalization ( E in E out ) An Effective Number of Hypotheses A Combinatorial Puzzle M. Magdon-Ismail CSCI 4100/6100 recap: The Two Questions

700 views • 18 slides
CSE 255 Lecture 2 Data Mining and Predictive Analytics Supervised learning Regression

CSE 255 Lecture 2 Data Mining and Predictive Analytics Supervised learning Regression

CSE 255 Lecture 2 Data Mining and Predictive Analytics Supervised learning Regression Supervised versus unsupervised learning Learning approaches attempt to model data in order to solve a problem Unsupervised learning approaches find

535 views • 50 slides
CSE 190 Lecture 2 Data Mining and Predictive Analytics Supervised learning Regression

CSE 190 Lecture 2 Data Mining and Predictive Analytics Supervised learning Regression

CSE 190 Lecture 2 Data Mining and Predictive Analytics Supervised learning Regression Supervised versus unsupervised learning Learning approaches attempt to model data in order to solve a problem Unsupervised learning approaches find

822 views • 50 slides
Cognitive Principles in Tutor & Cognitive Tutor Principles e-Learning Design

Cognitive Principles in Tutor & Cognitive Tutor Principles e-Learning Design

Lots of Lists of Principles 1 Cognitive Principles in Tutor & Cognitive Tutor Principles e-Learning Design Koedinger, K. R. & Corbett, A. T. (2006). Cognitive Tutors: Technology bringing learning science to the classroom.

325 views • 13 slides
Supervised Learning: The Setup Machine Learning 1 Last lecture We saw What is learning?

Supervised Learning: The Setup Machine Learning 1 Last lecture We saw What is learning?

Supervised Learning: The Setup Machine Learning 1 Last lecture We saw What is learning? Learning as generalization The badges game 2 This lecture More badges Formalizing supervised learning Instance space and features

1.04k views • 80 slides
Age Age Age Age E out ( h 1 ) E out ( h 2 ) E out ( h 3 ) E out ( h M )

Age Age Age Age E out ( h 1 ) E out ( h 2 ) E out ( h 3 ) E out ( h M )

recap: Verification h Income Learning From Data Lecture 4 Real Learning is Feasible Age Hoeffding: E out ( h ) E in ( h ) (with high probability) E out ( h ) Real Learning vs. Verification The Two Step Solution to Learning D Closer

240 views • 5 slides
Output g f after looking at data ( x n , y n ). 2. Can We do it? E in E out simple H ,

Output g f after looking at data ( x n , y n ). 2. Can We do it? E in E out simple H ,

recap: Three Learning Principles Scientist 1 Scientist 2 Scientist 3 resistivity resistivity resistivity Occams razor: simpler is better; falsifiable. Learning From Data temperature T temperature T temperature T Lecture 15 not

835 views • 3 slides
Learning Chapter 6 Learning Principles Denied existence of entity called personality

Learning Chapter 6 Learning Principles Denied existence of entity called personality

Learning Chapter 6 Learning Principles Denied existence of entity called personality Disregards unconscious motivation. Behavior can be controlled by its consequences, by reinforcer that follows the behavior. Learning Principles

522 views • 16 slides
Principles of Learning Assorted Principles for More Effective Training VP University

Principles of Learning Assorted Principles for More Effective Training VP University

1 Principles of Learning Assorted Principles for More Effective Training VP University Principles of Learning 2 Pluralistic Ignorance People are Afraid to Ask Questions Fancy word: Pluralistic ignorance A false impression of how other

193 views • 15 slides
DATA ANALYTICS USING DEEP LEARNING GT 8803 // FALL 2018 // NIDHI MENON LECTURE #15: THE DATA

DATA ANALYTICS USING DEEP LEARNING GT 8803 // FALL 2018 // NIDHI MENON LECTURE #15: THE DATA

DATA ANALYTICS USING DEEP LEARNING GT 8803 // FALL 2018 // NIDHI MENON LECTURE #15: THE DATA CALCULATOR: DATA STRUCTURE DESIGN AND COST SYNTHESIS FROM FIRST PRINCIPLES TODAYs PAPER The Data Calculator: Data Structure Design and Cost

522 views • 39 slides
RL LECTURE 3 SIMPLE LEARNING TAXONOMY LEARNING FROM INTERACTION Supervised Learning with

RL LECTURE 3 SIMPLE LEARNING TAXONOMY LEARNING FROM INTERACTION Supervised Learning with

RL LECTURE 3 SIMPLE LEARNING TAXONOMY LEARNING FROM INTERACTION Supervised Learning with environment Teacher provides required response to inputs. De- sired behaviour known. Costly to achieve some goal

298 views • 4 slides
Data Mining Learning from Large Data Sets Lecture 3

Data Mining Learning from Large Data Sets Lecture 3

Data Mining Learning from Large Data Sets Lecture 3 Locality Sensi7ve Hashing 263-5200-00L Andreas Krause Announcement No class next week

810 views • 47 slides
Data Mining Learning from Large Data Sets Lecture 2

Data Mining Learning from Large Data Sets Lecture 2

Data Mining Learning from Large Data Sets Lecture 2 Nearest neighbor search 263-5200-00L Andreas Krause Announcement Homework 1 out by

980 views • 56 slides

More recommend