Teaching Dimension COMS 6998-4 Learning Theory Benjamin Kuykendall brk2117@columbia.edu 1 November 2017 Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 1 / 38
Outline 1 Introduction Learning model Generic bounds 2 Examples Least Teachable Class Axis Aligned Boxes 3 Teaching versus Learning Disparities Bounds 4 Recursive Teaching Almost maximal Classes Recursive Teaching Dimension Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 2 / 38
Outline 1 Introduction Learning model Generic bounds 2 Examples Least Teachable Class Axis Aligned Boxes 3 Teaching versus Learning Disparities Bounds 4 Recursive Teaching Almost maximal Classes Recursive Teaching Dimension Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 3 / 38
Consistent learners and Helpful Directors [Goldman, Rivest, & Shapire 1993] Definition (Consistent learner) A learner is consistent when for all t the is some f ∈ C such that ∀ i < t , f ( x i ) = f ∗ ( x i ) f ( x t ) = y t and Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 4 / 38
Consistent learners and Helpful Directors [Goldman, Rivest, & Shapire 1993] Definition (Consistent learner) A learner is consistent when for all t the is some f ∈ C such that ∀ i < t , f ( x i ) = f ∗ ( x i ) f ( x t ) = y t and In the online model, after inputs x 1 , x 2 , ..., x i : No consistent learner will make a mistake at t > i ⇔ Exactly one consistent hypothesis is consistent with the x < t Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 4 / 38
Teaching dimension [Goldman & Kearns 1995] Definition (Teaching Sequence) Inputs x 1 ,..., x m are a teaching sequence for f when there is no other function g ∈ C such that g ( x i ) = f ( x i ) for all i ≤ m . Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 5 / 38
Teaching dimension [Goldman & Kearns 1995] Definition (Teaching Sequence) Inputs x 1 ,..., x m are a teaching sequence for f when there is no other function g ∈ C such that g ( x i ) = f ( x i ) for all i ≤ m . Definition (Teaching Dimension) The class C has teaching dimension of t when t is the smallest integer such that each f ∈ C has a teaching sequence of length at most t . Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 5 / 38
Outline 1 Introduction Learning model Generic bounds 2 Examples Least Teachable Class Axis Aligned Boxes 3 Teaching versus Learning Disparities Bounds 4 Recursive Teaching Almost maximal Classes Recursive Teaching Dimension Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 6 / 38
Trivial Teaching Sequence Theorem (Teaching Upper Bound) Any finite class has a teaching dimension at most t ≤ ∣C∣ − 1 . Enumerate C = f , f 1 ,..., f ∣C∣− 1 . To teach f , choose x i such that f ( x i ) ≠ f i ( x i ) . Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 7 / 38
Counting Teaching Sequences Theorem (Teaching Lower Bound) Any finite class C over X has a teaching dimension at least t ≤ log ∣C∣ − 1 log ∣ X ∣ . Each f uniquely identified by some x 1 ,..., x t with f ( x 1 ) ,..., f ( x t ) . ∣C∣ ≤ 2 t (∣ X ∣ t ) ≤ 2 ∣ X ∣ t . Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 8 / 38
Summary of Generic Bounds Theorem (Teaching Bounds) Any finite class C over X has a teaching dimension t such that ∣C∣ − 1 ≥ t ≥ log ∣C∣ − 1 log ∣ X ∣ . Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 9 / 38
Outline 1 Introduction Learning model Generic bounds 2 Examples Least Teachable Class Axis Aligned Boxes 3 Teaching versus Learning Disparities Bounds 4 Recursive Teaching Almost maximal Classes Recursive Teaching Dimension Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 10 / 38
Least Teachable Class Example (Least Teachable Class) Consider the following concept class over { 1 , 2 , ..., n } : C = { X ∖ { 1 } , X ∖ { 2 } ,..., X ∖ { n }} ∪ { X } . To teach X ∖ { i } use teaching sequence i . To teach X need sequence 1 , 2 , ..., n . So teaching dimension is n = ∣C∣ − 1. Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 11 / 38
Outline 1 Introduction Learning model Generic bounds 2 Examples Least Teachable Class Axis Aligned Boxes 3 Teaching versus Learning Disparities Bounds 4 Recursive Teaching Almost maximal Classes Recursive Teaching Dimension Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 12 / 38
Rectangles in the Plane Example (Rectangles in Z 2 ) Two points x , y ∈ Z 2 define a rectangle R x , y ( z ) = 1 ⇔ z 1 ∈ [ x 1 , y 1 ] and z 2 ∈ [ x 2 , y 2 ] . Teaching sequence Positive examples: x and y Negative examples: x − ( 1 , 0 ) , x − ( 0 , 1 ) , y + ( 1 , 0 ) , y + ( 0 , 1 ) Teaching dimension 6 Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 13 / 38
Rectangles in the Plane - - + - + - Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 14 / 38
Rectangles in the Plane - - + - + - Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 15 / 38
Higher Dimensions Example (Boxes in Z d ) Two points x , y ∈ Z d define a box R x , y ( z ) = 1 ⇔ ∀ i ∈ [ d ] z i ∈ [ x i , y i ] . Teaching sequence Positive examples: x and y Negative examples for each i ∈ [ d ] : x − e i , y + e i Teaching dimension 2 ( 1 + d ) Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 16 / 38
Unions of boxes Example (Union of Boxes) Fix k . For R x 1 , y 1 , ..., R x k , y k disjoint each in R d let k U { x i , y i } ( z ) = ⋃ R x i , y i . i = 1 Use the union of the teaching sequences for each box (with special case when boxes are adjacent) Teaching dimension 2 k ( 1 + d ) . Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 17 / 38
Union of boxes - - + - + - - + - - + - Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 18 / 38
Union of boxes ( k = 2) - - + - + - - + - - + - Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 19 / 38
Union of boxes ( k = ?) - - + - + - - + - - + - Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 20 / 38
Union of boxes ( k = 2) - - + - + - - + - - + - Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 21 / 38
Outline 1 Introduction Learning model Generic bounds 2 Examples Least Teachable Class Axis Aligned Boxes 3 Teaching versus Learning Disparities Bounds 4 Recursive Teaching Almost maximal Classes Recursive Teaching Dimension Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 22 / 38
VC Dimension Definition (Shattered set) The class C shatters a set S ⊂ X when { S ∩ c ∶ c ∈ C } = P ( S ) . Definition (VC dimension) The integer d is the Vapnik-Chervonenkis dimension of a class C if it is the minimum d such that C shatters no sets of d + 1 points. Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 23 / 38
Hard to teach, easy to learn Example (Least Teachable Class) C = { X ∖ { 1 } , X ∖ { 2 } ,..., X ∖ { n }} ∪ { X } . Teaching Dimension n VC Dimension 2 as no hypothesis induces ( 1 , 0 , 0 ) on three points Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 24 / 38
Infinite Teaching Dimension [Moran, Shpilka, Wigderson, Yehudayoff 2015] Example (Dedekind cuts) Consider the class of sets of rational numbers less than some real C = {(−∞ , r ) ∩ Q ∶ r ∈ R } . VC Dimension 2 as for q 1 < q 2 < q 3 cannot induce ( 1 , 0 , 1 ) Teaching Dimension ∞ Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 25 / 38
Easy to teach, hard to learn Set of n easy to teach functions: F = {{ x } ∶ x ∈ [ n ]} Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 26 / 38
Easy to teach, hard to learn Set of n easy to teach functions: F = {{ x } ∶ x ∈ [ n ]} Set of 2 m hard to learn functions: G = 2 [ m ] Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 26 / 38
Easy to teach, hard to learn Set of n easy to teach functions: F = {{ x } ∶ x ∈ [ n ]} Set of 2 m hard to learn functions: G = 2 [ m ] Choose 2 m = n and construct class over [ n ] ⊍ [ m ] Example (Hybrid Concept) Enumerate F = f 1 , ..., f n and G = g 1 , ..., g m above. Define class C = { h i = f i ⊍ g i ∶ i ∈ [ n ]} . Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 26 / 38
Easy to teach, hard to learn x 1 x 2 x 3 x n − 1 x n y 1 y m − 1 y m ... ... + − − − − − − − h 1 ... ... − + − − − − − + h 2 ... ... − − + − − − + − h 3 ... ... ⋮ − − − + − + + − h n − 1 ... ... − − − − + + + + h n ... ... Benjamin Kuykendall (brk2117) Teaching Dimension 1 November 2017 27 / 38
Recommend
More recommend
