BREAK POINTS BREAK POINTS Matthieu R Bloch Tuesday, March 3, 2020 - PowerPoint PPT Presentation

BREAK POINTS BREAK POINTS Matthieu R Bloch Tuesday, March 3, 2020 1

LOGISTICS LOGISTICS TAs and Office hours Tuesday: Dr. Bloch (College of Architecture Cafe) - 11am - 11:55am Tuesday: TJ (VL C449 Cubicle D) - 1:30pm - 2:45pm Thursday: Hossein (VL C449 Cubicle B): 10:45pm - 12:00pm Friday: Brighton (TSRB 523a) - 12pm-1:15pm Projects Proposals due Friday March 13, 2020 Midterm Thursday March 5th Sample midterm posted ( do not share ) Open notes but not open electronics (no space on the desks :() 2

RECAP: DICHOTOMY RECAP: DICHOTOMY The union bound naturally depends on the size of , but this is not what matters H We will consider the number of hypotheses that lead to distinct labeling for a dataset Definition. (Dichotomy) For a dataset and set of hypotheses , the set of dichotomies generated by on is D ≜ { x i } N H H D i =1 x i } N x i } N H ({ ) ≜ {{ h ( ) : h ∈ H } i =1 i =1 By definition and in general 2 N x i } N x i } N | H ({ )| ≤ | H ({ )| ≪ | H | i =1 i =1 Definition. (Growth function) For a set of hypotheses , the growth function of is H H x i } N m H ( N ) ≜ max | H ({ )| i =1 { x i } N i =1 The growth function does not depend on the datapoints and 2 N { x i } N m H ( N ) ≤ i =1 3

RECAP: EXAMPLES RECAP: EXAMPLES Positive rays: H ≜ { h : R → {±1} : x ↦ sgn ( x − a )| a ∈ R } m H = N + 1 Positive intervals: H ≜ { h : R → {±1} : x ↦ 1 { x ∈ [ a ; b ]} − 1 { x ∉ [ a ; b ]}| a < b ∈ R } N + 1 1 1 2 N 2 m H = ( ) + 1 = + N + 1 2 2 4

EXAMPLES EXAMPLES Convex sets: R 2 R 2 H ≜ { h : → {±1}|{ x ∈ : h ( x ) = +1} is convex } because we can generate all dichotomies 2 N m H ( N ) = Definition. (Shattering) If can generate all dichotomies on , we say that shatters { x i } N { x i } N H H i =1 i =1 5

6

EXAMPLES EXAMPLES Linear classifiers: R 2 R 2 w ⊺ H ≜ { h : → {±1} : x ↦ sgn ( x + b )| w ∈ , b ∈ R } The growth function is a worst case measure, hence m H (3) = 8 4 points cannot always be shattered and 2 4 m H (4) = 14 < 7

8

WISHFUL THINKING WISHFUL THINKING The growth function can sometimes be smaller than m H | H | What if we could use instead of in our analysis? m H | H | We know that with probability at least 1 − δ − − − − − − − − − − 1 2| H | h ∗ ˆ N h ∗ √ R ( ) ≤ R ( ) + log 2 N δ What if we could show that with probability at least 1 − δ − − − − − − − − − − − − − 1 2 m H ( N ) h ∗ ˆ N h ∗ √ R ( ) ≤ R ( ) + log ? 2 N δ Our ability to generalize then depend on the behavior of the growth function If is exponential in , we can’t say much m H ( N ) N If is polynomial in , then for large enough h ∗ ˆ N h ∗ m H ( N ) N R ( ) ≈ R ( ) N 9

10

BREAK POINTS BREAK POINTS Definition. (Break point) If not data set of size can be shattered by , then is a break point for k H k H If is a break point then 2 k k m H ( k ) < Examples Positive rays: break point k = 2 Postive Intervals: break point k = 3 Convex sets: break point k = ∞ Linear classifiers: break point k = 4 We will spend some time proving the following result Proposition. If there exists any break point for , then is polynomial in H m H ( N ) N If there is no break point for , then 2 N H m H ( N ) = 11

BREAK POINTS BREAK POINTS Definition. Assume has break point . is the maximum number of dichotomies of points such that H k B ( N , k ) N no subset of size can be shattered by the dichotomies. k is a purely combinatorial quantity, does not depend on B ( N , k ) H Example. Assume has break point . How many dichotomies can we generate of set of size 3? H 2 By definition, if is a break point for , then k H m H ( N ) ≤ B ( N , k ) 12

13

SAUER’S LEMMA SAUER’S LEMMA Lemma. , for , B ( N , 1) = 1 B (1, k ) = 2 k > 1 ∀ k > 1 B ( N , k ) ≤ B ( N − 1, k ) + B ( N − 1, k − 1) Lemma (Sauer's lemma) k −1 N B ( N , k ) ≤ ∑ ( ) i i =0 is polynomial in of degree B ( N , k ) N k − 1 Conclusion: if has a break point, is polynomial in H m H ( N ) N 14

15

16

   17