Statistical Classification with Fisher Zantedeschi Introduction - PowerPoint PPT Presentation

Statistical Classification with Fisher Kernel Valentina Statistical Classification with Fisher Zantedeschi Introduction Kernel Topic Models LDA PLSM Fisher Kernel Valentina Zantedeschi Results Supervisors: R´ emi Emonet, Marc Sebban September 3, 2014

Temporal documents classification Statistical Classification with Fisher Kernel Valentina Goal Zantedeschi improve discriminational power of topic models Introduction Topic Models LDA PLSM Fisher Kernel Approch Results learn topic models build a classifier based on fisher vector 2 / 22

Generative Topic Models Statistical Classification with Fisher Kernel Valentina Model extraction Zantedeschi find the set of topics that most probably had generated Introduction the observations Topic Models LDA PLSM Fisher Kernel 1 Latent Dirichlet Allocation : text Results documents, images 2 Probabilistic Latent Sequential Motifs : videos, sounds 3 / 22

Topic Models Classification Statistical Classification with Fisher Kernel Valentina Advantages Zantedeschi Introduction 1 lower dimensional Topic Models representation : noise LDA PLSM reduction, smaller Fisher Kernel datasets Results 2 captures the contest of words : detects synonyms and polysems 4 / 22

Latent Dirichlet Allocation Statistical Classification with Fisher Kernel Valentina Zantedeschi 1 I like eating broccoli and bananas Introduction 2 I ate a banana and spinach smoothie for breakfast Topic Models LDA PLSM 3 Chinchillas and kittens are cute Fisher Kernel 4 My sister adopted a kitten yesterday Results 5 Look at this cute hamster munching a piece of broccoli 5 / 22

Definitions and Assumptions Statistical Classification with Fisher Kernel Vocabulary Valentina Zantedeschi set of the possible values of the words Introduction Topic Models word LDA PLSM v 1 broccoli Fisher Kernel v 2 banana Results v 3 cute v 4 eat ... ... w = v 1 6 / 22

Definitions and Assumptions Statistical Classification with Fisher Kernel Topic Valentina Zantedeschi mixture of words : ∀ k , ∀ v Pr( w ji = v | z ji = k ) Introduction Topic Models LDA Topic A PLSM Fisher Kernel 30% broccoli, 15% banana, 10% breakfast, 10% munch, Results 0%cute Topic B 20% chinchilla, 20% kitten, 20% cute, 15% hamster, ... 7 / 22

Definitions and Assumptions Statistical Document Classification with Fisher Kernel Valentina A document d is a combination of words of the Zantedeschi vocabulary Introduction mixture of topics : ∀ w ji , ∀ k Pr( z ji = k ) = N d ( z ji = k ) Topic Models N d LDA PLSM Fisher Kernel 1 I like eating broccoli and bananas : 100% Topic A Results 2 I ate a banana and spinach smoothie for breakfast : 100% Topic A 3 Chinchillas and kittens are cute : 100% Topic B 4 My sister adopted a kitten yesterday : 100% Topic B 5 Look at this cute hamster munching on a piece of broccoli : 50% Topic A, 50% Topic B 8 / 22

A formal representation Statistical Classification with Fisher Kernel Valentina Zantedeschi Introduction w di : the term i of the document d Topic Models LDA z di : its topic PLSM Fisher Kernel Results θ dk = P ( z di = k ) φ kv = P ( w di = v | z di = k ) 9 / 22

Probabilistic Latent Sequential Motifs Statistical Classification with Fisher Kernel Valentina Zantedeschi Introduction t s : starting time Topic Models t a : absolute time LDA PLSM t r : relative time Fisher Kernel Results t a = t s + t r 10 / 22

An example of temporal document Statistical Classification with Fisher Kernel Valentina Zantedeschi Introduction Topic Models LDA PLSM Fisher Kernel Japanese Thrush Results Pre-processing : extracting words Mel-frequency cepstral coefficients (MFCC) : sound power distribution over frequences 11 / 22

Definitions and Assumptions Statistical Classification with Fisher Kernel Motifs Valentina Zantedeschi mixture of words in a temporal order: ∀ t r , ∀ w Pr( w , t r ) Introduction Topic Models LDA PLSM Fisher Kernel Results Yellowthroat 12 / 22

Definitions and Assumptions Statistical Document Classification with Fisher Kernel A document j is a combination of words of the Valentina Zantedeschi vocabulary in a temporal order Introduction mixtures of motifs starting at each instant: Topic Models LDA ∀ t s , ∀ z Pr( z , t s ) PLSM Fisher Kernel Results 13 / 22

Topic Models issues for classification Statistical Classification with Fisher Kernel Valentina Zantedeschi relevance of words combination Introduction Topic Models number of topics LDA PLSM Fisher Kernel Results We can do better.... 14 / 22

Similarity Statistical Classification with Fisher Kernel Valentina Zantedeschi Introduction Topic Models LDA PLSM Fisher Kernel Results 15 / 22

Similarity Statistical Classification with Fisher Kernel Valentina Zantedeschi Introduction Topic Models LDA PLSM Fisher Kernel Results 16 / 22

Fisher Kernel Statistical Classification with Fisher Kernel Valentina Fisher Score Zantedeschi Introduction U X = ∇ θ log Pr( X | θ ) Topic Models LDA PLSM Fisher Kernel Results Fisher Kernel T I − 1 U Y K ( X , Y ) = U X 17 / 22

Fisher Score for LDA Statistical Classification with θ k = P ( z i = k ) φ kr = P ( w i = r | z i = k ) Fisher Kernel Valentina Zantedeschi It combines the advantages of the BoW and Topic Model classifiers Introduction Topic Models ∂θ k = � V ∂ f v =1 n ( v )( C kv − θ k ) LDA PLSM Fisher Kernel � V ∂ f ∂φ kr = n ( r ) C kr − φ kr v =1 n ( v ) C kv Results It is more accurate It still works with small training datasets It works even with few topics 18 / 22

BoW / LDA / Fisher Score Statistical dataset size : 2000 documents Classification with Fisher Kernel proportion test documents / training documents : 10% Valentina Zantedeschi Introduction Feuille1 Topic Models Fisher Score Fisher Kernel BoW LDA Fisher Score LDA 97,74 45,19 98,3 PLSM 100 86,8 97,74 72,31 98,3 95 87,44 97,74 76,83 99,43 Fisher Kernel 86,13 97,74 77,81 97,74 90 88,13 97,74 82,48 97,74 Results 85 85,53 97,74 88,7 98,3 80 85,62 97,74 93,05 98,3 75 86,19 97,74 94,12 98,3 accuracy BoW 70 97,74 93,5 97,74 86,48 LDA 65 79,66 97,74 88,13 87 Fisher Score 60 73,44 97,74 83,61 82,48 72,31 97,74 87,71 90,96 55 77,96 97,74 86,32 89,83 50 79,09 97,74 83,61 85,87 45 40 1 2 3 4 5 6 7 8 9 10 11 12 13 14 topics 19 / 22

Fisher Score / Fisher Kernel Statistical dataset size : 2000 documents Classification with Fisher Kernel proportion test documents / training documents : 10% Valentina Zantedeschi Introduction Feuille1 Topic Models Fisher Score Fisher Kernel LDA 98,3 PLSM 100 98,3 86,8 99,43 87,44 Fisher Kernel 95 97,74 86,13 97,74 88,13 Results 90 98,3 85,53 98,3 85,62 85 98,3 86,19 accuracy 80 97,74 86,48 Fisher Score 87 79,66 Fisher Kernel 75 82,48 73,44 90,96 72,31 70 89,83 77,96 85,87 79,09 65 60 1 2 3 4 5 6 7 8 9 10 11 12 13 14 topics 20 / 22

BoW / LDA / Fisher Score Statistical dataset size : 20000 documents Classification with Fisher Kernel proportion test documents / training documents : 10% Valentina classes = 20 Zantedeschi Introduction Topic Models Feuille1 LDA PLSM BoW LDA Fisher Score 90,84 5 90,84 Fisher Kernel 100 90,84 6,05 84,35 90,84 10,8 84,4 90 Results 90,84 15 82,5 80 90,84 15,75 74,9 90,84 17,25 71 70 90,84 23,1 68,85 60 90,84 25 68,3 accuracy 90,84 27,6 69 BoW 50 90,84 25,6 71 LDA 40 90,84 29,25 73,2 Fisher Score 30 90,84 34,04 74,5 90,84 39,1 76 20 90,84 42,3 77 10 90,84 43 76 90,84 44 76 0 90,84 50 82,5 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 90,84 52 84,4 90,84 topics 56 84,35 90,84 59 86 21 / 22

Statistical Classification with Fisher Kernel Valentina Zantedeschi Introduction Topic Models LDA THANKS FOR YOUR ATTENTION! PLSM Fisher Kernel Results 22 / 22