Learning From Data Lecture 6 Bounding The Growth Function Bounding - PowerPoint PPT Presentation

Learning From Data Lecture 6 Bounding The Growth Function Bounding the Growth Function Models are either Good or Bad The VC Bound - replacing |H| with m H ( N ) M. Magdon-Ismail CSCI 4100/6100

recap: The Growth Function m H ( N ) A new measure for the diversity of a hypothesis set. H ( x 1 , . . . , x N ) = { ( h ( x 1 ) , . . . , h ( x N )) } The dichotomies ( N -tuples) H implements on x 1 , . . . , x N . H viewed through D H The growth function m H ( N ) considers the worst possible x 1 , . . . , x N . m H ( N ) = max x 1 ,..., x N |H ( x 1 , . . . , x N ) | . This lecture: Can we bound m H ( N ) by a polynomial in N ? Can we replace |H| by m H ( N ) in the generalization bound? M Bounding the Growth Function : 2 /31 � A c L Creator: Malik Magdon-Ismail Example growth functions − →

Example Growth Functions N 1 2 3 4 5 · · · 2-D perceptron 2 4 8 14 · · · 1-D pos. ray 2 3 4 5 · · · < 2 5 · · · 2-D pos. rectangles 2 4 8 16 • m H ( N ) drops below 2 N – there is hope . • A break point is any k for which m H ( k ) < 2 k . M Bounding the Growth Function : 3 /31 � A c L Creator: Malik Magdon-Ismail Quiz I − →

Pop Quiz I I give you a set of k ∗ points x 1 , . . . , x k ∗ on which H implements < 2 k ∗ dichotomys. (a) k ∗ is a break point. (b) k ∗ is not a break point. (c) all break points are > k ∗ . (d) all break points are ≤ k ∗ . (e) we don’t know anything about break points. M Bounding the Growth Function : 4 /31 � A c L Creator: Malik Magdon-Ismail Answer − →

Pop Quiz I I give you a set of k ∗ points x 1 , . . . , x k ∗ on which H implements < 2 k ∗ dichotomys. (a) k ∗ is a break point. (b) k ∗ is not a break point. (c) all break points are > k ∗ . (d) all break points are ≤ k ∗ . � (e) we don’t know anything about break points. M Bounding the Growth Function : 5 /31 � A c L Creator: Malik Magdon-Ismail Quiz II − →

Pop Quiz II For every set of k ∗ points x 1 , . . . , x k ∗ , H implements < 2 k ∗ dichotomys. (a) k ∗ is a break point. (b) k ∗ is not a break point. (c) all k ≥ k ∗ are break points. (d) all k < k ∗ are break points. (e) we don’t know anything about break points. M Bounding the Growth Function : 6 /31 � A c L Creator: Malik Magdon-Ismail Answer − →

Pop Quiz II For every set of k ∗ points x 1 , . . . , x k ∗ , H implements < 2 k ∗ dichotomys. � (a) k ∗ is a break point. (b) k ∗ is not a break point. � (c) all k ≥ k ∗ are break points. (d) all k < k ∗ are break points. (e) we don’t know anything about break points. M Bounding the Growth Function : 7 /31 � A c L Creator: Malik Magdon-Ismail Quiz III − →

Pop Quiz III To show that k is not a break point for H : (a) Show a set of k points x 1 , . . . x k which H can shatter. (b) Show H can shatter any set of k points. (c) Show a set of k points x 1 , . . . x k which H cannot shatter. (d) Show H cannot shatter any set of k points. (e) Show m H ( k ) = 2 k . M Bounding the Growth Function : 8 /31 � A c L Creator: Malik Magdon-Ismail Answer − →

Pop Quiz III To show that k is not a break point for H : � (a) Show a set of k points x 1 , . . . x k which H can shatter. overkill (b) Show H can shatter any set of k points. (c) Show a set of k points x 1 , . . . x k which H cannot shatter. (d) Show H cannot shatter any set of k points. � (e) Show m H ( k ) = 2 k . M Bounding the Growth Function : 9 /31 � A c L Creator: Malik Magdon-Ismail Quiz IV − →

Pop Quiz IV To show that k is a break point for H : (a) Show a set of k points x 1 , . . . x k which H can shatter. (b) Show H can shatter any set of k points. (c) Show a set of k points x 1 , . . . x k which H cannot shatter. (d) Show H cannot shatter any set of k points. (e) Show m H ( k ) > 2 k . M Bounding the Growth Function : 10 /31 � A c L Creator: Malik Magdon-Ismail Answer − →

Pop Quiz IV To show that k is a break point for H : (a) Show a set of k points x 1 , . . . x k which H can shatter. (b) Show H can shatter any set of k points. (c) Show a set of k points x 1 , . . . x k which H cannot shatter. � (d) Show H cannot shatter any set of k points. (e) Show m H ( k ) > 2 k . M Bounding the Growth Function : 11 /31 � A c L Creator: Malik Magdon-Ismail Combinatorial puzzle again − →

Back to Our Combinatorial Puzzle How many dichotomies can you list on 4 points so that no 2 is shattered. x 1 x 2 x 3 x 4 ◦ ◦ ◦ ◦ ◦ ◦ ◦ • ◦ ◦ • ◦ ◦ • ◦ ◦ • ◦ ◦ ◦ Can we add a 6th dichotomy? M Bounding the Growth Function : 12 /31 � A c L Creator: Malik Magdon-Ismail Can’t add a 6th dichotomy − →

Can’t Add A 6th Dichotomy x 1 x 2 x 3 x 4 ◦ ◦ ◦ ◦ ◦ ◦ ◦ • ◦ ◦ • ◦ ◦ • ◦ ◦ • ◦ ◦ ◦ ◦ • • ◦ M Bounding the Growth Function : 13 /31 � A c L Creator: Malik Magdon-Ismail B ( N, K ) − →

The Combinatorial Quantity B ( N, k ) How many dichotomies can you list on 4 points so that no 2 are shattered. ↑ ↑ N k B ( N, k ): Max. number of dichotomys on N points so that no k are shattered. x 1 x 2 x 3 x 1 x 2 x 3 x 4 ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ • ◦ ◦ ◦ • ◦ • ◦ ◦ ◦ • ◦ • ◦ ◦ ◦ • ◦ ◦ • ◦ ◦ ◦ B (3 , 2) = 4 B (4 , 2) = 5 M Bounding the Growth Function : 14 /31 � A c L Creator: Malik Magdon-Ismail B (4 , 3) − →

Let’s Try To Bound B (4 , 3) How many dichotomies can you list on 4 points so that no subset of 3 is shattered. x 1 x 2 x 3 x 4 ◦ ◦ ◦ ◦ ◦ ◦ ◦ • ◦ ◦ • ◦ ◦ • ◦ ◦ • ◦ ◦ ◦ ◦ ◦ • • ◦ • ◦ • • ◦ ◦ • ◦ • • ◦ • ◦ • ◦ • • ◦ ◦ M Bounding the Growth Function : 15 /31 � A c L Creator: Malik Magdon-Ismail Two kinds of dichotomys − →

Two Kinds of Dichotomys Prefix appears once or prefix appears twice. x 1 x 2 x 3 x 4 ◦ ◦ ◦ ◦ ◦ ◦ ◦ • ◦ ◦ • ◦ ◦ • ◦ ◦ • ◦ ◦ ◦ ◦ ◦ • • ◦ • ◦ • • ◦ ◦ • ◦ • • ◦ • ◦ • ◦ • • ◦ ◦ M Bounding the Growth Function : 16 /31 � A c L Creator: Malik Magdon-Ismail Reorder the dichotomys − →

Reorder the Dichotomys x 1 x 2 x 3 x 4 ◦ • • ◦ • ◦ • ◦ α α : prefix appears once • • ◦ ◦ β : prefix appears twice ◦ ◦ ◦ ◦ ◦ ◦ • ◦ β ◦ • ◦ ◦ B (4 , 3) = α + 2 β • ◦ ◦ ◦ ◦ ◦ ◦ • ◦ ◦ • • β ◦ • ◦ • • ◦ ◦ • M Bounding the Growth Function : 17 /31 � A c L Creator: Malik Magdon-Ismail Bound for α + β − →

First, Bound α + β x 1 x 2 x 3 x 4 ◦ • • ◦ • ◦ • ◦ α • • ◦ ◦ ◦ ◦ ◦ ◦ α + β ≤ B (3 , 3) ◦ ◦ • ◦ β ◦ • ◦ ◦ ↑ • ◦ ◦ ◦ A list on 3 points, with no 3 shattered (why?) ◦ ◦ ◦ • ◦ ◦ • • β ◦ • ◦ • • ◦ ◦ • M Bounding the Growth Function : 18 /31 � A c L Creator: Malik Magdon-Ismail Bound for β − →

Second, Bound β x 1 x 2 x 3 x 4 ◦ • • ◦ • ◦ • ◦ α • • ◦ ◦ ◦ ◦ ◦ ◦ β ≤ B (3 , 2) ◦ ◦ • ◦ β ↑ ◦ • ◦ ◦ • ◦ ◦ ◦ If 2 points are shattered, then using the mirror di- ◦ ◦ ◦ • chotomies you shatter 3 points (why?) ◦ ◦ • • β ◦ • ◦ • • ◦ ◦ • M Bounding the Growth Function : 19 /31 � A c L Creator: Malik Magdon-Ismail Combine the bounds − →

Combining to Bound α + 2 β x 1 x 2 x 3 x 4 ◦ • • ◦ B (4 , 3) = α + β + β • ◦ • ◦ α • • ◦ ◦ ≤ B (3 , 3) + B (3 , 2) ◦ ◦ ◦ ◦ ◦ ◦ • ◦ β ◦ • ◦ ◦ • ◦ ◦ ◦ The argument generalizes to ( N, k ) ◦ ◦ ◦ • ◦ ◦ • • β ◦ • ◦ • B ( N, k ) ≤ B ( N − 1 , k )+ B ( N − 1 , k − 1) • ◦ ◦ • M Bounding the Growth Function : 20 /31 � A c L Creator: Malik Magdon-Ismail Simple boundary cases − →

Boundary Cases: B ( N, 1) and B ( N, N ) k 1 2 3 4 5 6 · · · 1 1 2 1 3 B ( N, 1) = 1 (why?) 3 1 7 N B ( N, N ) = 2 N − 1 4 1 15 (why?) 5 1 31 6 1 63 . . ... . . . . M Bounding the Growth Function : 21 /31 � A c L Creator: Malik Magdon-Ismail Getting B (3 , 2) − →

Recursion Gives B ( N, k ) Bound B ( N, k ) ≤ B ( N − 1 , k ) + B ( N − 1 , k − 1) k 1 2 3 4 5 6 · · · 1 1 2 1 3 ց ↓ 3 1 4 7 N 4 1 15 5 1 31 6 1 63 . . . . . . . ... . . . . . . . . . . . . . . M Bounding the Growth Function : 22 /31 � A c L Creator: Malik Magdon-Ismail Filling the table − →

Recursion Gives B ( N, k ) Bound B ( N, k ) ≤ B ( N − 1 , k ) + B ( N − 1 , k − 1) k 1 2 3 4 5 6 · · · 1 1 2 1 3 3 1 4 7 N 4 1 5 11 15 5 1 6 16 26 31 6 1 7 22 42 57 63 . . . . . . . ... . . . . . . . . . . . . . . k − 1 � N � M Bounding the Growth Function : 23 /31 � � A c L Creator: Malik Magdon-Ismail B ( N, k ) ≤ − → i i =0