Feature Representation Vision BoWs and Beyond Praveen Krishnan - PowerPoint PPT Presentation

Feature Representation – Vision BoWs and Beyond Praveen Krishnan

Feature Representation in Vision  Low Level  Local Detectors and Descriptors  Bag of Words  Mid Level  Parts  Attributes  Hierarchical  Deep Representations

Low Level Vision  Bag of Visual Words (BoWs)  Visual Vocabulary  Vector Quantization  Spatial Verification  Inspirations from IR  Advanced coding and pooling schemes  Soft quantization  Higher order representation

Bag of Words (BoWs)

A quick walk through..  BoWs Image Bag

A quick walk through..  Origins in text processing Salton & McGill (1983) Slide: ICCV 2005 short course, L. Fei-Fei

A quick walk through..  Origins in texture recognition Julesz, 1981 Histogram Universal texton dictionary

A quick walk through..  BoWs Representation (i) Interest Point Detection (ii) Feature Extraction (iii) Vector Quantization Visual Vocabulary (iv) Coding and Pooling Figure Courtesy: Tsai‟12

Devil is in the details  Local detectors & descriptors  SIFT, HOG, LBP, …  Vocabulary  k-means, approximate k-means, GMM  Coding and Pooling  Histogram, kernel code books, sparse codes, LLC, Fisher kernels, Super Vectors, VLAD  Average, Max  Spatial Verification  Spatial pyramids, Min Hash, LLAH  Recognition & Retrieval  SVMs  Weighting schemes, query expansion, re-ranking etc.

Devil is in the details Assume dense sampling at  Local detectors & descriptors multiple scales  SIFT, HOG, LBP, …  Vocabulary  k-means, approximate k-means, GMM  Coding and Pooling  Histogram, kernel code books, sparse codes, LLC, Fisher kernels, Super Vectors, VLAD  Average, Max  Spatial Verification  Spatial pyramids, Min Hash, LLAH  Recognition & Retrieval  SVMs  Weighting schemes, query expansion, re-ranking etc.

Feature Extraction  Detection  Regular  Fei-Fei et. al.‟ 05  Bosh et. al. „‟06  Sparse or Interest point  Mikolajczyk et. al ‟05 Descriptor  Csurka et al. 2004  Description  SIFT – Lowe‟99  MSER – Matas et.al. ‟02  HoG – Dalal et.al „05 Descriptor  many more…

Visual Words/ Learning Visual Vocabulary Code Words  Partitioning the local descriptor space into informative regions.  Let be „N‟ Clustering SIFT descriptors from a subset of entire corpus. Visual vocabulary  k-means or Codebook  Minimize sum of squared Euclidean distances between points and their nearest cluster centers.  Here B is the codebook Image patch example Sivic et.al. ICCV‟05

Visual vocabulary  Issues  Size of vocabulary?  Too small: visual words not representative of all patches.  Too large: quantization artifacts , over fitting  Generative or discriminative learning?  Gaussian mixture models. (More later)  Computational Efficiency  Approximate k-means using randomized kd-trees. Phibin et.al . CVPR‟07  Hierarchical K-Means. Nister et.al. CVPR‟07 Nister et.al. CVPR‟07

Coding Codebook   Vector quantization  Assigns each feature to the nearest visual word in the vocabulary.  Hard quantization.

Pooling Codebook   Invariance to changes in position, lightning conditions.  Robustness to clutter  Compactness of representation  Types  Sum or average  Max

Pooling Codebook   Invariance to changes in position, lightning conditions.  Robustness to clutter  Compactness of representation  Types  Sum or average  Max There goes the geometry too 

Spatial Pooling  Pyramid Match Kernel  Weighted sum of histogram intersections at multiple resolutions.  More weightage for matches found at fine level than Pyramid Match Kernel, Grauman et.al. ICCV‟05 coarse level.  Used for matching in high dimensional spaces.  Spatial Pyramid Matching  Concatenate the histogram vectors at all pyramid levels. Spatial Pyramid Matching, Lazebnik et. al. CVPR‟06

Recognition & Retrieval  Recognition  Discriminative Methods  K-nearest neighbor  SVMs  Non-linear kernels  Generative Methods  Naïve Bayes.  Bayesian Models. (pLSA, LDA)  Ranking & Retrieval  Nearest neighbor search Agenda for this talk  Indexing

Ranking & Retrieval  Similarity measures  Cosine distance L1  L1 distance  Chi-square distance Chi-Square  Hellinger distance Hellinger Applies discount to large values

Ranking & Retrieval  Earth Mover‟s Distance (EMD)  Computes dissimilarity between distribution.  Let be the distribution with „m‟ elements and be the distribution with „n‟ elements. The flow F that minimizes the overall cost is given as:- Distance between element s i and q j Transportation problem

Ranking & Retrieval Cats and Dogs Database  Evaluation measures  Notations: TP-True positives; FP-False positives; TN-True negatives; FN-False negatives  Precision (P): Query  Recall (R):  F-measure:  Mean Average Precision (mAP) TP TP FP TP FP  Area under precision and recall curve TN FN

Re-ranking using Geometric Verification  Use the position and shape of the underlying features to improve retrieval quality. Both images have many matches – which is correct?  Estimate geometric transformation to remove outliers  Approaches:  RANSAC  Hough Transform Slide Credit: Cordelia Schmid

Re-ranking using Geometric Verification  Fitting an affine transformation  Assume we know the correspondences, how do we get the transformation? Slide Credit: Cordelia Schmid

Slide Credit: Kristen Grauman Re-ranking using Geometric Verification E.g. Fitting a Line  RANSAC (Fischler & Bolles, 1981): Randomly select  Randomly select a seed group of matches minimal subset of  Compute transformation from seed group points  Find inliers to this transformation  If the number of inliers is sufficiently large, re-compute least-squares estimate of Hypothesize a model transformation on all of the inliers  Keep the transformation with the largest number of inliers Repeat hypothesize ‐ and Select points consistent with Compute error verify loop model function

Inspirations from IR  Making faster  Inverted indexing  Reverse look up  Enables fast search by exploiting the sparse representation. #Images #Visual Words Image Courtesy : Jawahar et. al DAS‟14

Inspirations from IR  Weighting schemes  Zipf‟s Law: Frequency of any word is inversely proportional to its rank.  TF-IDF Weighting:  Stop Words: T op 5% of frequent visual words. Image Courtesy Wikipedia

Inspirations from IR  Improving the recall  Query expansion: Reformulating the query to increase its expressiveness. E.g. adding synonyms, jittering etc. Results … Spatial verification Query image New results Repeat New query Chum et.al., Total Recall, ICCV’07

Inspirations from IR  Query Expansion:  Baseline  Transitive closure expansion  Use of priority queue.  Average query expansion  Recursive average query expansion  Multiple image resolution expansion  Compute the median image resolution  Formulate query for other resolution bands. (0, 4/5) - (2/3, 3/2) - (5/4, infinity)  Do average query expansion for each band. Chum et.al., Total Recall, ICCV’07

Advanced coding schemes

Lost in Quantization  Hard quantization (VQ)  Issues  codeword uncertainty  codeword plausibility.

Modeling Uncertainty  Kernel code books  Allowing a degree of ambiguity in assigning code words from image features.  Uses kernel density estimation  Kernel size determine the amount of smoothing  Kernel shape is related to distance function  Kernel Codebook Gemert et al, Kernel Codebooks for Scene Categorization, ECCV2008

Modeling Uncertainty  Code word uncertainty  Code word plausibility Gemert et al, Kernel Codebooks for Scene Categorization, ECCV2008

Encoding – Sparse Coding  From VQ to SC Too restrictive. Relax by L1 norm !  Max Pooling Yang, e.t.al, CVPR‟09

Encoding – Sparse Coding  Sparse Coding  Fix V and solve for U [LASSO]  Fix U and solve for V [Least Square]  Linear Classification using SPM kernel Yang, e.t.al, CVPR‟09

Encoding  SC results tends to be local.  Locality more essential than sparsity ?  Local coordinate coding (LCC)  Locality-constrained Linear Coding (LLC)  Dropping the sparsity term and evoking the locality term explicitly  Here denotes element wise multiplication and d i is the locality adaptor that gives different weights to different basis vector as per similarity . Wang et. al., CVPR‟10

Encoding - LLC  Comparison with VQ and SC  Better reconstruction  Local smooth sparsity  Analytical solution Wang et. al., CVPR‟10

Interpretation so far…  Discover sub spaces  Geometry of data manifold • Each basis an “anchor • Each basis is a “direction” point” • Sparsity: each datum is a • Sparsity is induced by linear combination of only locality: each datum is a several bases. linear combination of • Related to topic model neighbor anchors. Slide Credit: Kai Yu