Computational Learning Theory: Positive and negative learnability - PowerPoint PPT Presentation

Computational Learning Theory: Positive and negative learnability results Machine Learning 1 Slides based on material from Dan Roth, Avrim Blum, Tom Mitchell and others

Computational Learning Theory • The Theory of Generalization • Probably Approximately Correct (PAC) learning • Positive and negative learnability results • Agnostic Learning • Shattering and the VC dimension 2

This lecture: Computational Learning Theory • The Theory of Generalization • Probably Approximately Correct (PAC) learning • Positive and negative learnability results • Agnostic Learning • Shattering and the VC dimension 3

𝑛 > 1 + ln 1 𝜗 ln 𝐼 𝜀 What can be learned General conjunctions are PAC learnable – 𝐼 = Number of conjunctions of 𝑜 variables = 3 ! ln 𝐼 = 𝑜 ln(3) – Number of examples needed 𝑛 > " # 𝑜 ln 3 + ln " $ 4

𝑛 > 1 + ln 1 𝜗 ln 𝐼 𝜀 What can be learned General conjunctions are PAC learnable – 𝐼 = Number of conjunctions of 𝑜 variables = 3 ! ln 𝐼 = 𝑜 ln(3) – Number of examples needed 𝑛 > " # 𝑜 ln 3 + ln " $ If we want to guarantee a 95% chance of learning a hypothesis of at least 90% • ! !" "."$ #$% !" & accuracy, with 𝑜 = 10 Boolean variables, we need m > = 140 %.$ examples If 𝑜 = 100 , this goes to 1129. (linearly increases with n) • Increasing the confidence to 99% will cost 1145 examples (logarithmic with 𝜀 ) • These results hold for any consistent learner 5

𝑛 > 1 + ln 1 𝜗 ln 𝐼 𝜀 What can be learned 𝑚 "" ∨ 𝑚 "6 ∨ 𝑚 "7 ∧ 𝑚 6" ∨ 𝑚 66 ∨ 𝑚 67 3-CNF Subset of CNFs: Each conjunct can have at most three literals (i.e a variable or its negation) 9

𝑛 > 1 + ln 1 𝜗 ln 𝐼 𝜀 What can be learned 𝑚 "" ∨ 𝑚 "6 ∨ 𝑚 "7 ∧ 𝑚 6" ∨ 𝑚 66 ∨ 𝑚 67 3-CNF Subset of CNFs: Each conjunct can have at most three literals (i.e a variable or its negation) 10

𝑛 > 1 + ln 1 𝜗 ln 𝐼 𝜀 What can be learned 𝑚 "" ∨ 𝑚 "6 ∨ 𝑚 "7 ∧ 𝑚 6" ∨ 𝑚 66 ∨ 𝑚 67 3-CNF Subset of CNFs: Each conjunct can have at most three literals (i.e a variable or its negation) What is the sample complexity? That is, if we had a consistent learner, how many examples will it need to guarantee PAC learnability? We need the size of the hypothesis space. How many 3CNFs are there? Number of conjuncts = • A 3-CNF is a conjunction with these many variables. • |H| = Number of 3-CNFs = • log(|H|) = O(n 3 ) • log(|H|) is polynomial in n ) the sample complexity is also polynomial in n 11

𝑛 > 1 + ln 1 𝜗 ln 𝐼 𝜀 What can be learned 𝑚 "" ∨ 𝑚 "6 ∨ 𝑚 "7 ∧ 𝑚 6" ∨ 𝑚 66 ∨ 𝑚 67 3-CNF Subset of CNFs: Each conjunct can have at most three literals (i.e a variable or its negation) What is the sample complexity? That is, if we had a consistent learner, how many examples will it need to guarantee PAC learnability? We need the size of the hypothesis space. How many 3CNFs are there? 2𝑜 ! Number of conjuncts = 𝑃 • • A 3-CNF is a conjunction with these many variables. |𝐼| = Number of 3-CNFs = 𝑃 2 "# ( • log |𝐼| = 𝑃(𝑜 ! ) • log(|H|) is polynomial in n ) the sample complexity is also polynomial in n 12

𝑛 > 1 + ln 1 𝜗 ln 𝐼 𝜀 What can be learned 𝑚 "" ∨ 𝑚 "6 ∨ 𝑚 "7 ∧ 𝑚 6" ∨ 𝑚 66 ∨ 𝑚 67 3-CNF Subset of CNFs: Each conjunct can have at most three literals (i.e a variable or its negation) What is the sample complexity? That is, if we had a consistent learner, how many examples will it need to guarantee PAC learnability? We need the size of the hypothesis space. How many 3CNFs are there? 2𝑜 ! Number of conjuncts = 𝑃 • • A 3-CNF is a conjunction with these many variables. |𝐼| = Number of 3-CNFs = 𝑃 2 "# ( • log |𝐼| = 𝑃(𝑜 ! ) • 17

Computational Learning Theory: Positive and negative learnability - PowerPoint PPT Presentation

Computational Learning Theory: Positive and negative learnability results Machine Learning 1 Slides based on material from Dan Roth, Avrim Blum, Tom Mitchell and others Computational Learning Theory The Theory of Generalization Probably

Aggregating Evidence about the Positive and Negative Effects of Treatments using a Computational

Computational Learning Theory: Probably Approximately Correct (PAC) Learning Machine Learning 1

Computational Learning Theory: An Analysis of a Conjunction Learner Machine Learning Slides

Computational Learning Theory: Agnostic Learning Machine Learning 1 Slides based on material

Computational Learning Theory: The Theory of Generalization Machine Learning 1 Slides based on

Computational Learning Theory: Shattering and VC Dimensions Machine Learning 1 Slides based on

Applying Computational Learning Theory to Software Testing Neil Walkinshaw Computational

Electronic Components Battery A portable power source that has a positive and negative.

Computational Learning Theory: Occams Razor Machine Learning 1 Slides based on material from

Remove negative environments Promote positive growth Reward good / positive and

Positive thinking about negative temperatures or, negative absolute temperatures: facts and myths

Multiple Stressors Multiple Uses Multiple Rules Solution: Synchronicity Positive

Statistical Natural Language Processing Outcome Whether a review is negative or positive:

The Positive and Negative Influence of Search Results on People's Decisions about the Efficacy

Computational Learning Theory For which tasks is successful learning possible? Under what

CSCE 478/878 Lecture 3: Computational Learning Theory Stephen D. Scott (Adapted from Tom

Computational Learning Theory Based on Machine Learning, T. Mitchell, McGRAW Hill, 1997, ch.

Computational Learning Theory 1 / 22 Decidability Computation Decidability which

Computational and Statistical Learning Theory TTIC 31120 Prof. Nati Srebro Lecture 7:

CSCE 478/878 Lecture 3: Computational Learning Theory Examines the worst-case minimum and

Positive Symptoms of Schizophrenia Negative Symptoms of Schizophrenia 1 Functional Status 100%

Dynamics of a growth model with negative and positive externalities A. Antoci, S. Borghesi, M.

Computational Learning Theory - HT 2018 formerly Advanced Machine Learning (HT 2017) Introduction

Signed numbers Goals unsigned numbers - non-negative integers signed numbers - positive/negative