MERCER THEOREM MERCER THEOREM Matthieu R Bloch Thursday, February - PowerPoint PPT Presentation

MERCER THEOREM MERCER THEOREM Matthieu R Bloch Thursday, February 13, 2020 1

LOGISTICS LOGISTICS TAs and Office hours Tuesday: TJ (VL C449 Cubicle D) - 1:30pm - 2:45pm Wednesday: Matthieu (TSRB 423) - 12:00:pm-1:15pm Thursday: Hossein (VL C449 Cubicle B): 10:45pm - 12:00pm Friday: Brighton (TSRB 523a) - 12pm-1:15pm Homework 3 Due Wednesday February 19, 2020 11:59pm EST (Friday February 126, 2020 for DL) Please include separate PDF with your plots and listings Make sure you show your work, don’t leave gaps in logic 2

CLARIFICATION NAIVE BAYES FOR BAG OF WORDS CLARIFICATION NAIVE BAYES FOR BAG OF WORDS Bag of word model Ignore positions of the words, focus on number of occurrences Remove uninformative words, e.g., “the,” “an”, “of” that do are unlikely to matter ( stop words ) Naive Bayes Assume all features are conditionally independent given the class Document classification Think of document as x = [word 1, word 2, ⋯ , word n ] ⊺ Dictionary contains words labeled by j For every class k P (word i = j | k ) = μ jk Likelihood of document in class is x k n μ N j μ 1 { x i = j } P ( x | k ) = ∏ ∏ = ∏ jk jk i =1 j j where is the number of occurrences of word in the document. N j j 3

RECOMMENDATION FOR HOMEWORK RECOMMENDATION FOR HOMEWORK Follow instructions (single PDF, cover page, etc.) Provide justifications You need to show that you understand the material, not that you can reproduce the notes As a rule, you need words and sentences around your math. I care more about seeing reasoning that seeing the final answer (which I already know!) You do not have to use LaTeX is you’re not comfortable Work on a separate sheet before writing your final solution Check your grade against my record - that’s why I email you We’re not immune against clerical errors or not finding your solution Please don’t tell me how many points you should get 4

PROJECTS PROJECTS Form teams of 3-5 and report on the shared spreadsheet Link on canvas later today Topic is up to you! You can explore papers or play with scikit Not need to invent new things, but I want to see some depth Consult with me if you have ideas. It’s ok to use something related to your research Tentative timeline Teams formed within next few days Project proposal early March Project report and deliverable late April 5

RECAP: KERNEL METHODS RECAP: KERNEL METHODS Implicitly define features through the choice of a kernel Definition. (Inner product kernel) An inner product kernel is a mapping for which there exists a Hilbert space and a R d R d k : × → R H mapping such that R d Φ : → H R d ∀ u , v ∈ k ( u , v ) = ⟨ Φ( u ), Φ( v ) ⟩ H Example. Quadratic kernel u ⊺ ) 2 k ( u , v ) = ( v Definition. (Positive semidefinite kernel) A function is a positive semidefinite kernel if R d R d k : × → R is symmetric, i.e., k k ( u , v ) = k ( v , u ) for all , the Gram matrix is positive semidefinite, i.e., { x i } N K i =1 x ⊺ Kx ≥ 0 with K = [ K i , j ] and K i , j ≜ k ( x i x j , ) 6

7

KERNEL METHODS - MERCER’S THEOREM KERNEL METHODS - MERCER’S THEOREM Theorem. A function is an inner product kernel if and only if is a positive semidefinite kernel. R d R d k : × → R k Examples of kernels Homogeneous polynomial kernel: with m ∈ N ∗ u ⊺ ) m k ( u , v ) = ( v Inhomogenous polynomial kernel: with , c > 0 m ∈ N ∗ u ⊺ ) m k ( u , v ) = ( v + c ∥ u − v ∥ 2 Radial basis function (RBF) kernel: with σ 2 k ( u , v ) = exp ( − ) > 0 2 σ 2 Question: How do we kernelize maximum margin optimization problem and solve efficiently? 8

9

   10