Food Recognition using Fusion of Classifiers based on CNNs Eduardo - PowerPoint PPT Presentation

ICIAP 2017 Food Recognition using Fusion of Classifiers based on CNNs Eduardo Aguilar, Marc Bolaños, and Petia Radeva Presented by: Petia Radeva

Contents • The problem of automatic food analysis • Convolutional Neural Network (CNN) • CNNs Fusion for Food analysis • Results • Conclusions

The obesity pandemic

How is today the food intake annotated? 24 hours dietary recall Food frequency questionnaire

What we propose about it? Automatic visual food recognition tools for dietary assessment.

Visual Food Analysis Food Recognition Food Ingredients Recognition Food Detection Food Segmentation for portion estimation Food Localization 19:28

What is on the market? 19:28 Mainly focused on weight reduction, not so healthy habits!

How many images should contain the real FoodImageNet? 200000 1000 200000 1000 150000 800 100000 1000 231 50000 600 0 231 400 200 0 FoodDB ImageNet 28000000 0 300000000 250000000 1400000 200000000 1500000 150000000 100000000 1000000 1400000 150000 50000000 0 500000 150000 0 19:28 FoodDB ImageNet

One is for sure, if there is a solution, it is highly probable to need Deep learning!

Difficulties Intra-class variations Inter-class similarities Mixed items Bad Labeled 19:28

CNN Models Inception model 19:28

Decision Profile Construction 0.63 0.35 0.02 0.55 0.31 0.14 19:28

Decision Profile Construction 1 2 …. C classes [ [ …………. 0.63 0.35 0.02 1 ………….. …………… …………… 2 … . K 0.55 0.31 0.14 models 19:28

What about the class images? [ [ [ [ [ 0.63 0.35 0.02 [ [ 0.63 0.35 0.02 [ Mean 0.55 0.31 0.14 0.63 0.35 0.02 0.63 0.35 0.02 0.55 0.31 0.14 ……………. ……………. 0.55 0.31 0.14 0.55 0.31 0.14 [ [ 0.61 0.29 0.1 ……………. ……………. 0.51 0.34 0.15 Decision template construction DT i 19:28

Decision template classification [ [ 0.63 0.35 0.02 …………. DT 1 0.55 0.31 0.14 [ ? ? [ [ [ 0.63 0.35 0.02 0.61 0.29 0.1 ………… …………….. …………….. DT 2 0.55 0.31 0.14 0.51 0.34 0.15 [ [ DT C DP 0.63 0.35 0.02 …………. 0.55 0.31 0.14 19:28

How to compare DP to DT c? ? [ [ [ [ 0.63 0.35 0.02 0.61 0.29 0.1 ……………….. ……………….. ……………… ……………… 0.55 0.31 0.14 0.51 0.34 0.15 DT c DP 19:28

Distance functions DT c DP Euclidean distance-based similarity: 0.61 0.29 0.1 0.63 0.35 0.02 ……………… ……………… ……………….. ……………….. ………………… …................. ∑ 1 N ( A , B ) = 1 − i , j ( a i , j − b i , j ) 2 K * C 1 ( A , B ) = || A � B || S || ς || = 1 n Σ µ ς ( u i ) || A � B || where ||.|| is the relative cardinality of the fuzzy set. C 1 C 2 C 3 …. 19:28

New similarity functions DT c DP The consistensy index is defined as: 0.63 0.35 0.02 0.61 0.29 0.1 C ( A , B ) = sup u { µ A � B } ……………… ……………… ……………….. ……………….. …................. ………………… Similarity based on symmetric distance : 2 ( A , B ) = 1 − || A ∇ B || S A ∇ B where is the symmetric difference defined by the C 1 C 2 C 3 …. Hamming distances: :: µ A ∇ B = | µ A ( u ) − µ B ( u ) |

Method Overview

Validation: Food-11 dataset Classes : Bread, Dairy Products, Dessert, Egg, Fried food, Meat, Noodles_Pasta, Rice, Seafood, Soup, Vegetable_Fruit. 11 categories: meat, dessert, vegetales, pasta, rise, seafood, etc. Dataset #Classes #Train #Val #Test Balanced Food-11 11 9,866 3,430 3,347

Validation: Food-101 dataset 101 most frequent food in foodspotting.com: baby back ribs, chocolate cake, hot and sour soup, caesar salad, eggs benedict, etc. Dataset #Classes #Train #Val #Test Balanced Food-101 101 75,750 25,250 - X

Datasets Food-11: #Classes 11 #Train 9,866 #Val 3,430 #Test 3,347 Food-101: Balanced #Classes 101 #Train 75,750 #Val - #Test 25,250

Validation on Food101 Overall accuracy for different CNN models InceptionV3(3) InceptionV3(2) InceptionV3(1) ResNet50(3) s3 ResNet50(2) ResNet50(1) s2 80 82 84 86 s1 6 models n 5 models 4 models i3 3 models 2 models i2 i1 Relation btw # of models and c accuracy in Food101 85,5 86 86,5 87 87,5 88 88,5 89

Validation on Food11 Accuracy of single models s3 GoogLeNet(2) s2 ResNet50(2) s1 ResNet50(1) 89 90 91 92 93 94 n Series3 i5 Series2 Series1 i3 i2 i1 c Relation btw # of models and 93,4 93,6 93,8 94 94,2 94,4 94,6 94,8 95 accuracy in Food11 19:28

Results - Misclassified on Food-11 19:28

LogMeal App & Api Demo 19:28 Food category and class recognition

Conclusions • Automatic food analysis is a necessity to help healthy habits • We proposed a CNNs Fusion approach that improves the classification performance with respect to using CNN models separately. • CNNs Fusion outperforms in more that 10% the overall accuracy on Food-11. • Using our approach with the best models is highly likely to improve the performance on Food-101.