computational learning theory an analysis of a
play

Computational Learning Theory: An Analysis of a Conjunction Learner - PowerPoint PPT Presentation

Oct 10, 2023 •347 likes •845 views

Computational Learning Theory: An Analysis of a Conjunction Learner Machine Learning Slides based on material from Dan Roth, Avrim Blum, Tom Mitchell and others 1 This lecture: Computational Learning Theory The Theory of Generalization

  1. Computational Learning Theory: An Analysis of a Conjunction Learner Machine Learning Slides based on material from Dan Roth, Avrim Blum, Tom Mitchell and others 1

  2. This lecture: Computational Learning Theory • The Theory of Generalization • Probably Approximately Correct (PAC) learning • Positive and negative learnability results • Agnostic Learning • Shattering and the VC dimension 2

  3. Where are we? • The Theory of Generalization – When can be trust the learning algorithm? – What functions can be learned? – Batch Learning • Probably Approximately Correct (PAC) learning • Positive and negative learnability results • Agnostic Learning • Shattering and the VC dimension 3

  4. This section 1. Analyze a simple algorithm for learning conjunctions 2. Define the PAC model of learning 3. Make formal connections to the principle of Occam’s razor 4

  5. This section 1. Analyze a simple algorithm for learning conjunctions 2. Define the PAC model of learning 3. Make formal connections to the principle of Occam’s razor 5

  6. Learning Conjunctions The true function f = x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 Training data – <(1,1,1,1,1,1,…,1,1), 1> – <(1,1,1,0,0,0,…,0,0), 0> – <(1,1,1,1,1,0,...0,1,1), 1> – <(1,0,1,1,1,0,...0,1,1), 0> – <(1,1,1,1,1,0,...0,0,1), 1> – <(1,0,1,0,0,0,...0,1,1), 0> – <(1,1,1,1,1,1,…,0,1), 1> – <(0,1,0,1,0,0,...0,1,1), 0> 6

  7. Learning Conjunctions f = x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 Training data – <(1,1,1,1,1,1,…,1,1), 1> A simple learning algorithm ( Elimination ) – <(1,1,1,0,0,0,…,0,0), 0> – <( 1 , 1 , 1 , 1 , 1 ,0,...0,1, 1 ), 1> Discard all negative examples • – <(1,0,1,1,1,0,...0,1,1), 0> – <( 1 , 1 , 1 , 1 , 1 ,0,...0,0, 1 ), 1> – <(1,0,1,0,0,0,...0,1,1), 0> – <( 1 , 1 , 1 , 1 , 1 ,1,…,0, 1 ), 1> – <(0,1,0,1,0,0,...0,1,1), 0> 7

  8. Learning Conjunctions f = x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 Training data – <( 1 , 1 , 1 , 1 , 1 ,1,…,1, 1 ), 1> A simple learning algorithm ( Elimination ) – <(1,1,1,0,0,0,…,0,0), 0> – <( 1 , 1 , 1 , 1 , 1 ,0,...0,1, 1 ), 1> Discard all negative examples • – <(1,0,1,1,1,0,...0,1,1), 0> Build a conjunction using the features • – <( 1 , 1 , 1 , 1 , 1 ,0,...0,0, 1 ), 1> that are common to all positive – <(1,0,1,0,0,0,...0,1,1), 0> conjunctions – <( 1 , 1 , 1 , 1 , 1 ,1,…,0, 1 ), 1> h = x 1 ∧ x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 – <(0,1,0,1,0,0,...0,1,1), 0> 8

  9. Learning Conjunctions f = x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 Training data – <( 1 , 1 , 1 , 1 , 1 ,1,…,1, 1 ), 1> A simple learning algorithm ( Elimination ) – <(1,1,1,0,0,0,…,0,0), 0> – <( 1 , 1 , 1 , 1 , 1 ,0,...0,1, 1 ), 1> Discard all negative examples • – <(1,0,1,1,1,0,...0,1,1), 0> Build a conjunction using the features • – <( 1 , 1 , 1 , 1 , 1 ,0,...0,0, 1 ), 1> that are common to all positive – <(1,0,1,0,0,0,...0,1,1), 0> conjunctions – <( 1 , 1 , 1 , 1 , 1 ,1,…,0, 1 ), 1> h = x 1 ∧ x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 – <(0,1,0,1,0,0,...0,1,1), 0> Positive examples eliminate irrelevant features 9

  10. Learning Conjunctions f = x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 Training data – <( 1 , 1 , 1 , 1 , 1 ,1,…,1, 1 ), 1> A simple learning algorithm: – <(1,1,1,0,0,0,…,0,0), 0> – <( 1 , 1 , 1 , 1 , 1 ,0,...0,1, 1 ), 1> Discard all negative examples • – <(1,0,1,1,1,0,...0,1,1), 0> Build a conjunction using the features • – <( 1 , 1 , 1 , 1 , 1 ,0,...0,0, 1 ), 1> that are common to all positive – <(1,0,1,0,0,0,...0,1,1), 0> conjunctions – <( 1 , 1 , 1 , 1 , 1 ,1,…,0, 1 ), 1> h = x 1 ∧ x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 – <(0,1,0,1,0,0,...0,1,1), 0> Clearly this algorithm produces a conjunction that is consistent with the data, that is err S (h) = 0, if the target function is a monotone conjunction Exercise: Why? 10

  11. Learning Conjunctions: Analysis h = x 1 ∧ x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 f = x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 Claim 1: Any hypothesis consistent with the training data will only make mistakes on positive future examples Why? 11

  12. Learning Conjunctions: Analysis h = x 1 ∧ x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 f = x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 Claim 1: Any hypothesis consistent with the training data will only make mistakes on positive future examples Why? A mistake will occur only if some literal z (in our example x 1 ) is present in h but not in f This mistake can cause a positive example to be predicted as negative by h Specifically: x 1 = 0, x 2 =1, x 3 =1, x 4 =1, x 5 =1, x 100 =1 The reverse situation can never happen For an example to be predicted as positive in the training set, every relevant literal must have been present 12

  13. Learning Conjunctions: Analysis h = x 1 ∧ x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 f = x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 Claim 1: Any hypothesis consistent with the training data will only make mistakes on positive future examples Why? A mistake will occur only if some literal z (in our example x 1 ) is present in h but not in f This mistake can cause a positive example to be predicted as negative by h Specifically: x 1 = 0, x 2 =1, x 3 =1, x 4 =1, x 5 =1, x 100 =1 The reverse situation can never happen For an example to be predicted as positive in the training set, every relevant literal must have been present 13

  14. Learning Conjunctions: Analysis h = x 1 ∧ x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 f = x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 Claim 1: Any hypothesis consistent with the training data will only make mistakes on positive future examples - Why? f h + + - A mistake will occur only if some literal z (in our - example x 1 ) is present in h but not in f This mistake can cause a positive example to be predicted as negative by h Specifically: x 1 = 0, x 2 =1, x 3 =1, x 4 =1, x 5 =1, x 100 =1 The reverse situation can never happen For an example to be predicted as positive in the training set, every relevant literal must have been present 14

  15. Learning Conjunctions: Analysis Theorem: Suppose we are learning a conjunctive concept with n dimensional Boolean features using m training examples. If then, with probability > 1 - ± , the error of the learned hypothesis err D (h) will be less than ² . 15

  16. Learning Conjunctions: Analysis Theorem: Suppose we are learning a conjunctive concept with n dimensional Boolean features using m training examples. If Poly in n, 1/ ± , 1/ ² then, with probability > 1 - ± , the error of the learned hypothesis err D (h) will be less than ² . If we see these many training examples, then the algorithm will produce a conjunction that, with high probability, will make few errors 16

  17. Learning Conjunctions: Analysis Theorem: Suppose we are learning a conjunctive concept with n dimensional Boolean features using m training examples. If then, with probability > 1 - ± , the error of the learned hypothesis err D (h) will be less than ² . Let’s prove this assertion 17

  18. Proof Intuition h = x 1 ∧ x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 f = x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 What kinds of examples would drive a hypothesis to make a mistake? Positive examples, where x 1 is absent f would say true and h would say false None of these examples appeared during training Otherwise x 1 would have been eliminated If they never appeared during training, maybe their appearance in the future would also be rare! Let’s quantify our surprise at seeing such examples 18

  19. Proof Intuition h = x 1 ∧ x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 f = x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 What kinds of examples would drive a hypothesis to make a mistake? Positive examples, where x 1 is absent f would say true and h would say false None of these examples appeared during training Otherwise x 1 would have been eliminated If they never appeared during training, maybe their appearance in the future would also be rare! Let’s quantify our surprise at seeing such examples 19

  20. Proof Intuition h = x 1 ∧ x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 f = x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 What kinds of examples would drive a hypothesis to make a mistake? Positive examples, where x 1 is absent f would say true and h would say false None of these examples appeared during training Otherwise x 1 would have been eliminated If they never appeared during training, maybe their appearance in the future would also be rare! Let’s quantify our surprise at seeing such examples 20

  21. Proof Intuition h = x 1 ∧ x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 f = x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 What kinds of examples would drive a hypothesis to make a mistake? Positive examples, where x 1 is absent f would say true and h would say false None of these examples appeared during training Otherwise x 1 would have been eliminated If they never appeared during training, maybe their appearance in the future would also be rare! Let’s quantify our surprise at seeing such examples 21

Recommend

Applying Computational Learning Theory to Software Testing Neil Walkinshaw Computational

Applying Computational Learning Theory to Software Testing Neil Walkinshaw Computational

Applying Computational Learning Theory to Software Testing Neil Walkinshaw Computational Learning Theory (CLT) Theoretical analysis of questions of learnability . Is it feasible to learn a given type of concept from a set of examples?

1.14k views • 4 slides
Computational Learning Theory: Probably Approximately Correct (PAC) Learning Machine Learning 1

Computational Learning Theory: Probably Approximately Correct (PAC) Learning Machine Learning 1

Computational Learning Theory: Probably Approximately Correct (PAC) Learning Machine Learning 1 Slides based on material from Dan Roth, Avrim Blum, Tom Mitchell and others Computational Learning Theory The Theory of Generalization

413 views • 28 slides
CSCE 478/878 Lecture 3: Computational Learning Theory Stephen D. Scott (Adapted from Tom

CSCE 478/878 Lecture 3: Computational Learning Theory Stephen D. Scott (Adapted from Tom

CSCE 478/878 Lecture 3: Computational Learning Theory Stephen D. Scott (Adapted from Tom Mitchells slides) September 8, 2003 1 Introduction Combines machine learning with: Algorithm design and analysis Computational complexity

393 views • 27 slides
CSCE 478/878 Lecture 3: Computational Learning Theory Examines the worst-case minimum and

CSCE 478/878 Lecture 3: Computational Learning Theory Examines the worst-case minimum and

Introduction Combines machine learning with: Algorithm design and analysis Computational complexity CSCE 478/878 Lecture 3: Computational Learning Theory Examines the worst-case minimum and maximum data and time requirements for

222 views • 7 slides
Computational Learning Theory: Positive and negative learnability results Machine Learning 1

Computational Learning Theory: Positive and negative learnability results Machine Learning 1

Computational Learning Theory: Positive and negative learnability results Machine Learning 1 Slides based on material from Dan Roth, Avrim Blum, Tom Mitchell and others Computational Learning Theory The Theory of Generalization Probably

534 views • 40 slides
Computational Learning Theory: Agnostic Learning Machine Learning 1 Slides based on material

Computational Learning Theory: Agnostic Learning Machine Learning 1 Slides based on material

Computational Learning Theory: Agnostic Learning Machine Learning 1 Slides based on material from Dan Roth, Avrim Blum, Tom Mitchell and others This lecture: Computational Learning Theory The Theory of Generalization Probably

653 views • 44 slides
Computational Learning Theory: The Theory of Generalization Machine Learning 1 Slides based on

Computational Learning Theory: The Theory of Generalization Machine Learning 1 Slides based on

Computational Learning Theory: The Theory of Generalization Machine Learning 1 Slides based on material from Dan Roth, Avrim Blum, Tom Mitchell and others Checkpoint: The bigger picture Supervised learning: instances, concepts, and

895 views • 42 slides
Computational Learning Theory: Shattering and VC Dimensions Machine Learning 1 Slides based on

Computational Learning Theory: Shattering and VC Dimensions Machine Learning 1 Slides based on

Computational Learning Theory: Shattering and VC Dimensions Machine Learning 1 Slides based on material from Dan Roth, Avrim Blum, Tom Mitchell and others This lecture: Computational Learning Theory The Theory of Generalization Probably

1.09k views • 81 slides
Computational Learning Theory: Occams Razor Machine Learning 1 Slides based on material from

Computational Learning Theory: Occams Razor Machine Learning 1 Slides based on material from

Computational Learning Theory: Occams Razor Machine Learning 1 Slides based on material from Dan Roth, Avrim Blum, Tom Mitchell and others This lecture: Computational Learning Theory The Theory of Generalization Probably

585 views • 35 slides
Computational Learning Theory For which tasks is successful learning possible? Under what

Computational Learning Theory For which tasks is successful learning possible? Under what

Computational Learning Theory For which tasks is successful learning possible? Under what conditions is successful learning guaranteed? What is successful learning? Probably approximately correct (PAC) framework Bounds on number

422 views • 27 slides
Computational Learning Theory Based on Machine Learning, T. Mitchell, McGRAW Hill, 1997, ch.

Computational Learning Theory Based on Machine Learning, T. Mitchell, McGRAW Hill, 1997, ch.

0. Computational Learning Theory Based on Machine Learning, T. Mitchell, McGRAW Hill, 1997, ch. 7 Acknowledgement: The present slides are an adaptation of slides drawn by T. Mitchell 1. Main Questions in Computational Learning Theory

682 views • 42 slides
Computational Learning Theory 1 / 22 Decidability Computation Decidability which

Computational Learning Theory 1 / 22 Decidability Computation Decidability which

Computational Learning Theory 1 / 22 Decidability Computation Decidability which problems have algorithmic solutions Machine Learning Feasibility what assumptions must we make to trust that we can learn an unknown target

246 views • 22 slides
Computational and Statistical Learning Theory TTIC 31120 Prof. Nati Srebro Lecture 7:

Computational and Statistical Learning Theory TTIC 31120 Prof. Nati Srebro Lecture 7:

Computational and Statistical Learning Theory TTIC 31120 Prof. Nati Srebro Lecture 7: Computational Complexity of Learning Hardness of Improper Learning (continued) Agnostic Learning Hardness of Learning via Crypto Easy to generate

433 views • 24 slides
Computational Learning Theory - HT 2018 formerly Advanced Machine Learning (HT 2017) Introduction

Computational Learning Theory - HT 2018 formerly Advanced Machine Learning (HT 2017) Introduction

Computational Learning Theory - HT 2018 formerly Advanced Machine Learning (HT 2017) Introduction and Course Details Varun Kanade University of Oxford January 15, 2018 What is Machine Learning? 200-basic level categories Here: Six

466 views • 18 slides
Statistical and Computational Statistical and Computational Learning Theory Learning Theory

Statistical and Computational Statistical and Computational Learning Theory Learning Theory

Statistical and Computational Statistical and Computational Learning Theory Learning Theory Fundamental Question: Predict Error Rates Fundamental Question: Predict Error Rates Given: Given: The space H of hypotheses The space H of

497 views • 29 slides
Computational Learning Theory [read Chapter 7] [Suggested exercises: 7.1, 7.2, 7.5, 7.8]

Computational Learning Theory [read Chapter 7] [Suggested exercises: 7.1, 7.2, 7.5, 7.8]

Computational Learning Theory [read Chapter 7] [Suggested exercises: 7.1, 7.2, 7.5, 7.8] Computational learning theory Setting 1: learner p oses queries to teac her Setting 2: teac her c ho oses examples

491 views • 24 slides
Computational Learning Theory - MT 2018 Introduction and Course Details Varun Kanade University

Computational Learning Theory - MT 2018 Introduction and Course Details Varun Kanade University

Computational Learning Theory - MT 2018 Introduction and Course Details Varun Kanade University of Oxford October 8, 2018 What is Machine Learning? 200-basic level categories Here: Six pictures containing airplanes and people

330 views • 18 slides
Learning Equivalence Structures Luca San Mauro (Vienna University of Technology) Logic Colloquium

Learning Equivalence Structures Luca San Mauro (Vienna University of Technology) Logic Colloquium

Learning Equivalence Structures Luca San Mauro (Vienna University of Technology) Logic Colloquium 2018 joint work with Ekaterina Fokina and Timo Koetzing Computational Learning Theory Computational Learning Theory (CLT) is a vast research

900 views • 69 slides
Machine Learning for Computational Linguistics Some probability distributions .

Machine Learning for Computational Linguistics Some probability distributions .

Machine Learning for Computational Linguistics Some probability distributions . ltekin, SfS / University of Tbingen April 19/21, 2016 5 / 48 Probability theory Information theory 0 0 1 0 0 Probability mass function Example:

507 views • 7 slides
11 { &lt; t (0), t (1), t ( n -1)&gt; Correct Concept: Learn a decent approximation most of t (

11 { &lt; t (0), t (1), t ( n -1)&gt; Correct Concept: Learn a decent approximation most of t (

Learning Theory Theorems that characterize classes of learning problems or specific algorithms in terms of computational complexity or sample complexity , i.e. the number of training examples CS 391L: Machine Learning: necessary or

450 views • 7 slides
Auditory Scene Analysis: phenomena, theories and computational models July 1998 Dan Ellis

Auditory Scene Analysis: phenomena, theories and computational models July 1998 Dan Ellis

Auditory Scene Analysis: phenomena, theories and computational models July 1998 Dan Ellis International Computer Science Institute, Berkeley CA &lt;dpwe@icsi.berkeley.edu&gt; Outline 1 The computational theory of ASA 2 Cues &amp; grouping

636 views • 39 slides
Computational Logic: (Constraint) Logic Programming Theory, practice, and implementation Program

Computational Logic: (Constraint) Logic Programming Theory, practice, and implementation Program

Computational Logic: (Constraint) Logic Programming Theory, practice, and implementation Program Analysis, Debugging, and Optimization A Tour of ciaopp : The Ciao Prolog Preprocessor Department of Artificial Intelligence School of Computer

694 views • 30 slides
Peer Prediction Mechanisms and their Connections to Machine Learning Jens Witkowski ETH

Peer Prediction Mechanisms and their Connections to Machine Learning Jens Witkowski ETH

Peer Prediction Mechanisms and their Connections to Machine Learning Jens Witkowski ETH Zurich Game Theory Meets Computational Learning Theory Dagstuhl June 22, 2017 Joint work with David C. Parkes (Harvard) and Rafael Frongillo

831 views • 16 slides
Cubical Type Theory Goal: provide a computational justification for Homotopy Type Theory and

Cubical Type Theory Goal: provide a computational justification for Homotopy Type Theory and

Cubical Type Theory: a constructive interpretation of the univalence axiom Anders M ortberg Inria Sophia-Antipolis TTT: Type Theory Based Tools January 15, 2017 - 1 / 39 Cubical Type Theory Goal: provide a computational justification for

743 views • 39 slides

More recommend