extended bag of words formalism for image classification
play

Extended Bag-of-Words Formalism for Image Classification Sandra - PowerPoint PPT Presentation

Jun 01, 2023 •476 likes •1.3k views

Extended Bag-of-Words Formalism for Image Classification Sandra Avila 1 , 2 (Cotutelle PhD Candidate), ujo 1 (Advisor), Matthieu Cord 2 (Advisor), Arnaldo de A. Ara Nicolas Thome 2 (Co-Advisor), Eduardo Valle 3 (Collaborator) 1 Federal

  1. Extended Bag-of-Words Formalism for Image Classification Sandra Avila 1 , 2 (Cotutelle PhD Candidate), ujo 1 (Advisor), Matthieu Cord 2 (Advisor), Arnaldo de A. Ara´ Nicolas Thome 2 (Co-Advisor), Eduardo Valle 3 (Collaborator) 1 Federal University of Minas Gerais, NPDI Lab – UFMG, Belo Horizonte, Brazil 2 Pierre and Marie Curie University, UPMC-Sorbonne Universities, LIP6, Paris, France 3 State University of Campinas, RECOD Lab, FEEC – UNICAMP, Campinas, Brazil Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 1 / 56

  2. Image Classification: Why do we care? Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 2 / 56

  3. Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 3 / 56

  4. Huge amount of image is available Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 4 / 56

  5. Why image classification is a hard problem? Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 5 / 56

  6. Many classes and concepts Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 6 / 56

  7. Viewpoint changes Illumination variations Occlusion Background clutter Inter-class similarity Intra-class diversity Much diversity in the data Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 6 / 56

  8. How do we classify images? Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 7 / 56

  9. Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 8 / 56

  10. Problem Statement Given an image dataset, how to represent their visual content information for a classification task? Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 9 / 56

  11. Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 10 / 56

  12. night scenes sunset scenes young people old people Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 11 / 56

  13. Bag-of-Visual-Words ( BoW ) [Sivic and Zisserman, 2003; Csurka et al., 2004] Slide credit: Ken Chatfield Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 12 / 56

  14. Low-level Visual Feature Extraction patch 1  l 1 , 1 . . . l 1 ,N  l 2 , 1 . . . l 2 ,N    . .  . .   . .   l M, 1 . . . l M,N patch M Local feature extraction Patch detection : interest points, dense sampling, . . . Feature extraction : SIFT [Lowe, 2004], SURF [Bay et al., 2008], . . . Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 13 / 56

  15. Visual Codebook Coding step Visual codebook learning : random, unsupervised (e.g., k -means, GMM), supervised [Perronnin et al., 2006; Goh et al., 2012], . . . Coding : hard-assignment, soft-assignment [van Gemert et al., 2008, 2010], sparse coding [Yang et al., 2009; Boureau et al., 2010], . . . Feature coding based on the vector difference : VLAD [J´ egou et al., 2010], SVC [Zhou et al., 2010], VLAT [Picard et al., 2011], . . . Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 14 / 56

  16. Pooling step Pooling : sum/average-pooling, max-pooling [Yang et al., 2009], . . . Spatial pooling : spatial pyramid matching [Lazebnik et al., 2006], [Jia et al., 2012], . . . Spatial Pyramid Matching Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 15 / 56

  17. Other Approaches Biologically-inspired Models Deep Learning Models [Fukushima and Miyake, 1982; LeCun et al., [Hinton and Salakhutdinov, 2006; 1990; Riesenhuber and Poggio, 1999; Serre Ranzato et al., 2007; Bengio, 2009] et al., 2007; Th´ eriault et al., 2012] Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 16 / 56

  18. BossaNova Representation Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 17 / 56

  19. Coding & Pooling Matrix Representation ... ... x 1 x j x N   α 1 , 1 . . . α 1 ,j . . . α 1 ,N c 1 . . . . . . . . . . . .     H = α m, 1 . . . α m,j . . . α m,N   c m   . . . . . . . .   . . . .   α M, 1 . . . α M,j . . . α M,N c M Notations : X = { x j } , j ∈ { 1 , . . . , N } : set of local descriptors (e.g., SIFT) C = { c m } , m ∈ { 1 , . . . , M } : visual codebook Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 18 / 56

  20. Coding & Pooling Matrix Representation ... ... x 1 x j x N   α 1 , 1 . . . α 1 ,j . . . α 1 ,N c 1 . . . . . . . . . . . .     H = α m, 1 . . . α m,j . . . α m,N   c m   . . . . . . . .   . . . .   α M, 1 . . . α M,j . . . α M,N c M ⇓ f : Coding � x j − c k � 2 Coding : x j → f ( x j ) = { α m,j } , α m,j = 1 iff m = arg min 2 k ∈{ 1 ,...,M } Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 18 / 56

  21. Coding & Pooling Matrix Representation ... ... x 1 x j x N   α 1 , 1 . . . α 1 ,j . . . α 1 ,N c 1 . . . . . . . . . . . .     H = α m, 1 . . . α m,j . . . α m,N ⇒ g : Pooling   c m   . . . . . . . .   . . . .   α M, 1 . . . α M,j . . . α M,N c M � x j − c k � 2 Coding : x j → f ( x j ) = { α m,j } , α m,j = 1 iff m = arg min 2 k ∈{ 1 ,...,M } N � Pooling : g ( { α j } ) = z : ∀ m, z m = α m,j j =1 Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 18 / 56

  22. Coding & Pooling Matrix Representation ... ... x 1 x j x N     z 1 α 1 , 1 . . . α 1 ,j . . . α 1 ,N c 1 . . . . . . . . . . . . . . .         z = H = α m, 1 . . . α m,j . . . α m,N z m     c m     . . . . . . . . . .     . . . . .     z M α M, 1 . . . α M,j . . . α M,N c M � x j − c k � 2 Coding : x j → f ( x j ) = { α m,j } , α m,j = 1 iff m = arg min 2 k ∈{ 1 ,...,M } N � Pooling : g ( { α j } ) = z : ∀ m, z m = α m,j j =1 BoW representation : z = [ z 1 , z 2 , · · · , z M ] T Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 18 / 56

  23. Early Ideas We pointed out the weakness in the standard pooling operation used in the BoW signature generation. Instead of averaging all the values from one row in the H matrix, we proposed to describe their distribution. BOSSA representation ( B ag O f S tatistical S ampling A nalysis) introduces our density function-based pooling strategy . Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 19 / 56

  24. Early Ideas We pointed out the weakness in the standard pooling operation used in the BoW signature generation. Instead of averaging all the values from one row in the H matrix, we proposed to describe their distribution. BOSSA representation ( B ag O f S tatistical S ampling A nalysis) introduces our density function-based pooling strategy . Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 19 / 56

  25. Early Ideas We pointed out the weakness in the standard pooling operation used in the BoW signature generation. Instead of averaging all the values from one row in the H matrix, we proposed to describe their distribution. BOSSA representation ( B ag O f S tatistical S ampling A nalysis) introduces our density function-based pooling strategy . Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 19 / 56

  26. Our Pooling Illustration Our Pooling BoW Pooling Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 20 / 56

  27. Our Pooling Formalism g : ❘ N ❘ B − → α m − → g ( α m ) = z m � b B ; b + 1 � �� z m,b = card x j | α m,j ∈ B b and b + 1 B ≥ α min ≤ α max m m B B denotes the number of bins of each histogram z m , and [ α min m ; α max ] limits the range of distances m Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 21 / 56

  28. BossaNova Representation Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 22 / 56

  29. BossaNova Representation ... ... x 1 x j x N   α 1 , 1 . . . α 1 ,j . . . α 1 ,N c 1 . . . . . . . . . . . .   exp − β m d 2 ( x j , c m )   α m,j = α m, 1 . . . α m,j . . . α m,N   c m � K   m ′ =1 exp − β m d 2 ( x j , c m ′ ) . . . . . . . .   . . . .   α M, 1 . . . α M,j . . . α M,N c M Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 22 / 56

  30. BossaNova Representation Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 22 / 56

  31. BossaNova Representation Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 22 / 56

  32. BossaNova Representation   z 1 , st 1 . . .     z m , st m     . .   .   z M , st M Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 22 / 56

  33. BossaNova Scheme Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 23 / 56

  34. BossaNova Scheme • SIFT descriptors on a dense spatial grid at multiple scales • Dimensionality reduction by applying PCA (128 → 64) Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 23 / 56

  35. BossaNova Scheme • k -means algorithm Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 23 / 56

  36. BossaNova Scheme Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 23 / 56

  37. BossaNova Scheme • SVM classifiers are applied by using a nonlinear Gauss– ℓ 2 kernel Sandra Avila (UFMG/UPMC) sandra@dcc.ufmg.br June 2013 23 / 56

Recommend

Bag of Words Model Overview of todays lecture Bag-of-words. K-means clustering.

Bag of Words Model Overview of todays lecture Bag-of-words. K-means clustering.

Bag of Words Model Overview of todays lecture Bag-of-words. K-means clustering. Classification. K nearest neighbors. Support vector machine. Image Classification Image Classification: Problem Data-driven approach

1.52k views • 104 slides
Classification Image Classification Set of predefined categories [eg: table, apple, dog, giraffe]

Classification Image Classification Set of predefined categories [eg: table, apple, dog, giraffe]

Day 1 Lecture 2 Classification Image Classification Set of predefined categories [eg: table, apple, dog, giraffe] Binary classification [1, 0] DOG 2 Image Classification 3 Image Classification pipeline Dog 4 Slide credit: Jose M lvarez

339 views • 23 slides
From image classification to object detection Image classification Object detection Image source

From image classification to object detection Image classification Object detection Image source

From image classification to object detection Image classification Object detection Image source Slides from L. Lazebnik What are the challenges of object detection? Images may contain more than one class, multiple instances from the same

1.11k views • 57 slides
Image Analysis System Example: Image Classification System pre feature feature segmentation

Image Analysis System Example: Image Classification System pre feature feature segmentation

Image Analysis System Example: Image Classification System pre feature feature segmentation processing description selection post feature classification classification extraction 1 IIP Image Segmentation Grey-level thresholding

423 views • 15 slides
for Image Classification Qilong Wang ( ) Dalian University of Technology

for Image Classification Qilong Wang ( ) Dalian University of Technology

Codebook-free Single Gaussian for Image Classification Qilong Wang ( ) Dalian University of Technology http://ice.dlut.edu.cn/PeihuaLi/ Image Model for Classification Scene Object Image Model Fine-grained Texture Face

539 views • 43 slides
Object Detection JunYoung Gwak 1 Motivation Image classification Input: Image

Object Detection JunYoung Gwak 1 Motivation Image classification Input: Image

Object Detection JunYoung Gwak 1 Motivation Image classification Input: Image Output: object class 2 Motivation Limitation of classification Multiple classes Location i.e. Object classification assumes Single

1.07k views • 50 slides
NO WORDS.... please use this image (turn horizontal?):

NO WORDS.... please use this image (turn horizontal?):

NO WORDS.... please use this image (turn horizontal?): https://unsplash.com/photos/4rDCa5hBlCs NO WORDS.... Please use this image https://unsplash.com/photos/ePvBOHsAA54 NO WORDS.... PLease use this image

472 views • 26 slides
Tuning the Performance of Convolutional Neural Network for Image Classification on GPU Agenda

Tuning the Performance of Convolutional Neural Network for Image Classification on GPU Agenda

Tuning the Performance of Convolutional Neural Network for Image Classification on GPU Agenda Adoptions of Image classification or image recognition at Alibaba Easy ways to improve performance of Caffe Further performance

182 views • 15 slides
Natural Language Processing Classification I Dan Klein UC Berkeley 1 2 Classification

Natural Language Processing Classification I Dan Klein UC Berkeley 1 2 Classification

Natural Language Processing Classification I Dan Klein UC Berkeley 1 2 Classification Classification Automatically make a decision about inputs Example: document category Example: image of digit digit Example: image

757 views • 50 slides
1 Image Classification BVM 2018 Tutorial: Advanced Deep Learning Methods Jakob Wasserthal,

1 Image Classification BVM 2018 Tutorial: Advanced Deep Learning Methods Jakob Wasserthal,

1 Image Classification BVM 2018 Tutorial: Advanced Deep Learning Methods Jakob Wasserthal, Division of Medical Image Computing Author Division Classification of skin cancer 02.11.16 | vs Esteva et al., Dermatologist-level classification of

1.08k views • 48 slides
for Large-Scale Image Classification Karn Simonyan, Andrea Vedaldi, Andrew Zisserman Visual

for Large-Scale Image Classification Karn Simonyan, Andrea Vedaldi, Andrew Zisserman Visual

Deep Fisher Networks for Large-Scale Image Classification Karn Simonyan, Andrea Vedaldi, Andrew Zisserman Visual Geometry Group, University of Oxford Deep learning achieves excellent performance in image classification. Do hand-crafted image

159 views • 4 slides
Gaze Embeddings for Zero-Shot Image Classification Nour Karessli Zeynep Akata Bernt

Gaze Embeddings for Zero-Shot Image Classification Nour Karessli Zeynep Akata Bernt

Gaze Embeddings for Zero-Shot Image Classification Nour Karessli Zeynep Akata Bernt Schiele Andreas Bulling Presentation by Hsin-Ping Huang and Shubham Sharma Introduction Attributes Standard image classification models fail

758 views • 28 slides
Prospects with Extended RPA Theories P. Papakonstantinou Institut f ur Kernphysik,

Prospects with Extended RPA Theories P. Papakonstantinou Institut f ur Kernphysik,

Prospects with Extended RPA Theories P. Papakonstantinou Institut f ur Kernphysik, T.U.Darmstadt 1 Overview Introduction From RPA to Second RPA - Formalism and technicalities - Results on Giant Resonances - Issues to be considered

412 views • 40 slides
Metric Learning Applied for Automatic Large Image Classification Supervisors TOON CALDERS (PhD) /

Metric Learning Applied for Automatic Large Image Classification Supervisors TOON CALDERS (PhD) /

September, 2014 UPC Metric Learning Applied for Automatic Large Image Classification Supervisors TOON CALDERS (PhD) / ULB SAHILU WENDESON / IT4BI SALIM JOUILI (PhD) /EuraNova Image Database Classification How? Using K-Nearest Neighbor (kNN)

415 views • 40 slides
Hybrid Deep Learning Topology for Image Classification Petru Radu petru.radu@ness.com 27 th

Hybrid Deep Learning Topology for Image Classification Petru Radu petru.radu@ness.com 27 th

15.07.2019 Hybrid Deep Learning Topology for Image Classification Petru Radu petru.radu@ness.com 27 th Summer School on Image Processing, Timisoara 2019 SSIP 2019 Introduction Classical Neural Networks employed in image classification have a

513 views • 33 slides
Statistical NLP Spring 2011 Lecture 11: Classification Dan Klein UC Berkeley Classification

Statistical NLP Spring 2011 Lecture 11: Classification Dan Klein UC Berkeley Classification

Statistical NLP Spring 2011 Lecture 11: Classification Dan Klein UC Berkeley Classification Automatically make a decision about inputs Example: document category Example: image of digit digit Example: image of object

609 views • 48 slides
using formalism in HCI from cognitive models to placemats From Formalism to

using formalism in HCI from cognitive models to placemats From Formalism to

using formalism in HCI from cognitive models to placemats From Formalism to Physicality, Alan Dix, UPC North, 30 April 2008 what to model users cognitive models task models system behaviour

314 views • 5 slides
Image Classification with Fashion-MNIST and CIFAR-10 Khoi Hoang California State University,

Image Classification with Fashion-MNIST and CIFAR-10 Khoi Hoang California State University,

Image Classification with Fashion-MNIST and CIFAR-10 Khoi Hoang California State University, Sacramento Problem Statement Image Classification is one of the most fundamental problem in the field of machine learning. Has many area of

639 views • 10 slides
a brief history of formalism from Aristotle to Alan Turing From Formalism to

a brief history of formalism from Aristotle to Alan Turing From Formalism to

a brief history of formalism from Aristotle to Alan Turing From Formalism to Physicality, Alan Dix, UPC North, 30 April 2008 first steps Aristotle (384 BC - 322 BC) foundations of logic Euclid (325 BC - 265 BC)

217 views • 4 slides
Bag-of-features models for category classification for category classification Cordelia Schmid

Bag-of-features models for category classification for category classification Cordelia Schmid

Bag-of-features models for category classification for category classification Cordelia Schmid Category recognition Category recognition Image classification: assigning a class label to the image Image classification: assigning a class

998 views • 75 slides
PLT Project SIP(Simplified Image Processing) A Language for image processing Why SIP ??

PLT Project SIP(Simplified Image Processing) A Language for image processing Why SIP ??

PLT Project SIP(Simplified Image Processing) A Language for image processing Why SIP ?? Effectively an image processing language. Concept can be extended for videos Features included to make operations on images short and effective

733 views • 12 slides
Lecture 7: Multiclass Classification Princeton University COS 495 Instructor: Yingyu Liang

Lecture 7: Multiclass Classification Princeton University COS 495 Instructor: Yingyu Liang

Machine Learning Basics Lecture 7: Multiclass Classification Princeton University COS 495 Instructor: Yingyu Liang Example: image classification indoor Indoor outdoor Example: image classification (multiclass) ImageNet figure borrowed from

807 views • 28 slides
AMMI Introduction to Deep Learning 7.2. Networks for image classification Fran cois

AMMI Introduction to Deep Learning 7.2. Networks for image classification Fran cois

AMMI Introduction to Deep Learning 7.2. Networks for image classification Fran cois Fleuret https://fleuret.org/ammi-2018/ Thu Sep 6 15:26:25 CAT 2018 COLE POLYTECHNIQUE FDRALE DE LAUSANNE Image classification, standard convnets

827 views • 71 slides
Generative Adversarial Networks Phillip Isola 9.520 10/17/18 Image classification Classifier

Generative Adversarial Networks Phillip Isola 9.520 10/17/18 Image classification Classifier

z N ( ~ 0 , 1) Generative Adversarial Networks Phillip Isola 9.520 10/17/18 Image classification Classifier Fish image X label Y Image generation Generator Fish label Y image X <latexit

1.3k views • 106 slides

More recommend