Spatial and Temporal representations for Multi-Modal Visual - PowerPoint PPT Presentation

Spatial and Temporal representations for Multi-Modal Visual Retrieval 17th December 2018 Noa Garcia Docampo PhD Candidate, Aston University

Introduction Million of images created every day... Million of images created every day... Problem : How to find images in large Problem : How to find images in large collections? collections?

Introduction Million of images created every day... Million of images created every day... Problem : How to find images in large Problem : How to find images in large collections? collections? Solution : Visual Retrieval! Image Retrieval exists from the 90s ● Many types of visual retrieval ●

Introduction We classify visual retrieval into 3 main types, depending on the query object and the dataset content:

Structure Introduction and Background S y m m e t r i c V R i s e u t r a i l e v a l Asymmetric Visual Retrieval Cross-Modal Retrieval Conclusions and Final Remarks

Contributions CNNs for non-metric visual similarity ● Symmetric Pushing performance on standard CBIR datasets Visual Retrieval ● MoviesDB: image-to-video retrieval dataset ● Binary descriptors for local aggregation of video features ● Asymmetric Spatio-temporal encoders for global aggregation of video features Visual Retrieval ● Item video retrieval application ● SemArt: semantic art understanding dataset ● Cross-Modal Cross-modal retrieval for semantic art understanding Retrieval ●

Introduction and Background S y m m e t r i c V R i s e u t r a i l e v a l

Symmetric Visual Retrieval Standard CBIR system

Symmetric Visual Retrieval Drawbacks of metric distances Do not consider data distribution ● Standard CBIR system

Symmetric Visual Retrieval Drawbacks of metric distances Do not consider data distribution ● Metric distance constraints: ● ○ ○ ○ ○ Standard CBIR system

Symmetric Visual Retrieval Drawbacks of metric distances Do not consider data distribution ● Metric distance constraints: ● Standard CBIR system

Symmetric Visual Retrieval Standard CBIR system Proposed CBIR system Garcia & Vogiatzis (2018). Learning Non-Metric Visual Similarity for Image Retrieval. Under review at IMAVIS journal

Similarity Networks

Symmetric Visual Retrieval Off-the-shelf methods Garcia & Vogiatzis (2018). Learning Non-Metric Visual Similarity for Image Retrieval. Under review at IMAVIS journal

Symmetric Visual Retrieval Off-the-shelf methods Fine-tuned methods Garcia & Vogiatzis (2018). Learning Non-Metric Visual Similarity for Image Retrieval. Under review at IMAVIS journal

Contributions CNNs for non-metric visual similarity ● Symmetric Pushing performance on standard CBIR datasets ● Visual Retrieval

Introduction and Background S y m m e t r i c V R i s e u t r a i l e v a l Asymmetric Visual Retrieval

Asymmetric Visual Retrieval Garcia & Vogiatzis (2018). Temporal Aggregation of Visual Features for Large-Scale Image-to-Video Retrieval. In: ICMR 2018

Asymmetric Visual Retrieval Chapter 5 Chapter 6 No temporal aggregation Garcia & Vogiatzis (2018). Temporal Aggregation of Visual Features for Large-Scale Image-to-Video Retrieval. In: ICMR 2018

Asymmetric Visual Retrieval Temporal Local Aggregation Feature Indexing Garcia & Vogiatzis (2018). Dress like a Star: Retrieving Fashion Products from Videos. In: CVF workshop ICCV 2017

Asymmetric Visual Retrieval Temporal Local Aggregation Search and Retrieval Garcia & Vogiatzis (2018). Dress like a Star: Retrieving Fashion Products from Videos. In: CVF workshop ICCV 2017

Asymmetric Visual Retrieval Chapter 5 Chapter 6 No temporal aggregation

Asymmetric Visual Retrieval Spatio-Temporal Global Aggregation Garcia & Vogiatzis (2018). Asymmetric Spatio-Temporal Embeddings for Large-Scale Image-to-Video Retrieval. In: BMVC 2018

Asymmetric Visual Retrieval Spatio-Temporal Global Aggregation Chapter 6 Temporal Local Aggregation Chapter 5 ● High accuracy ● Global aggregation state-of-the-art accuracy ● High compression rates ● High compression rates ● Multiple searches per query ● Single search per query

Contributions CNNs for non-metric visual similarity ● Symmetric Pushing performance on standard CBIR datasets ● Visual Retrieval MoviesDB: image-to-video retrieval dataset ● Binary descriptors for local aggregation of video features ● Asymmetric Spatio-temporal encoders for global aggregation of video features Visual Retrieval ● Item video retrieval application ●

Introduction and Background S y m m e t r i c V R i s e u t r a i l e v a l Asymmetric Visual Retrieval Cross-Modal Retrieval

Cross-Modal Retrieval Retrieve paintings from artistic comments Artistic Comments: ● Not only descriptions of the content but also ○ about the author, context, techniques, etc. Fine-art paintings: ● ○ Figurative representations Garcia & Vogiatzis (2018). How to Read Paintings: Semantic Art Understanding with Multi-Modal Retrieval. In: VISART workshop ECCV 2018

Cross-Modal Retrieval Visual Encoding (images): VGG16, ResNet , RMAC ● Text Encoding (comments and titles): BOW , MLP, RNN ● Cross-Modal Transformation: CCA, Cosine Margin Loss , Augmented with Metadata ● Garcia & Vogiatzis (2018). How to Read Paintings: Semantic Art Understanding with Multi-Modal Retrieval. In: VISART workshop ECCV 2018

Cross-Modal Retrieval Same type images Random images Human Comparison: Easy Set Human Comparison: Difficult Set Garcia & Vogiatzis (2018). How to Read Paintings: Semantic Art Understanding with Multi-Modal Retrieval. In: VISART workshop ECCV 2018

Contributions CNNs for non-metric visual similarity ● Symmetric Pushing performance on standard CBIR datasets ● Visual Retrieval MoviesDB: image-to-video retrieval dataset ● Binary descriptors for local aggregation of video features ● Asymmetric Spatio-temporal encoders for global aggregation of video features Visual Retrieval ● Item video retrieval application ● SemArt: semantic art understanding dataset ● Cross-Modal Cross-modal retrieval for semantic art understanding Retrieval ●

Introduction and Background S y m m e t r i c V R i s e u t r a i l e v a l Asymmetric Visual Retrieval Cross-Modal Retrieval Conclusions and Final Remarks

Future Work Symmetric Similarity networks for other retrieval tasks ● Visual Retrieval Temporal aggregation at the scene level ● Asymmetric Visual Retrieval Asymmetric techniques for video-to-image retrieval ● Style and content detector for cross-modal retrieval in art ● Cross-Modal Retrieval SemArt dataset for alternative tasks ●

Introduction and Background S y m m e t r i c V R i s e u t r a i l e v a l

Content-Based Image Retrieval

Similarity Networks Input : Concatenation of feature vectors ● Architecture : Fully connected layers with ReLU ● Output : Similarity score ● Loss Function Network Output

Similarity Networks Input : Concatenation of feature vectors ● Architecture : Fully connected layers with ReLU ● Output : Similarity score ● Loss Function Pair Label

Similarity Networks Input : Concatenation of feature vectors ● Architecture : Fully connected layers with ReLU ● Output : Similarity score ● Loss Function Margin

Similarity Networks Input : Concatenation of feature vectors ● Architecture : Fully connected layers with ReLU ● Output : Similarity score ● Loss Function Standard Similarity

Spatial and Temporal representations for Multi-Modal Visual - PowerPoint PPT Presentation

Spatial and Temporal representations for Multi-Modal Visual Retrieval 17th December 2018 Noa Garcia Docampo PhD Candidate, Aston University Introduction Million of images created every day... Million of images created every day... Problem :

Temporal and Modal Logic Based on paper: E.A. Emerson. Temporal and Modal Logic J. van Leeuwen,

Commercial Detection in Heterogeneous Video Streams Using Fused Multi-Modal and Temporal Features

Multi-modal Face Presentation Attack Detection via Spatial and Channel Attentions Guoqing Wang 1,3

Temporal, Spatial, and Spatio-temporal Granularities Gabriele Pozzani Department of Computer

Outline Introduction Background and Motivation Spatial, Temporal and Spatiotemporal

Spatial navigation in humans Recap: navigation strategies and spatial representations Spatial

Hierarchical Spatial Gossip for Hierarchical Spatial Gossip for Multi- -Resolution

Introduction to Temporal Logic Mehdi Dastani BBL-521 M.M.Dastani@uu.nl Modal Logic Modal logic

Statistical Methods for Multivariate Spatial and Spatial-Temporal Processes Mikyoung Jun

Using R for time series analysis and spatial-temporal distribution of global burnt surface

Multi-modal Face Recognition Hu Han hanhu@ict.ac.cn http: / / vipl.ict.ac.cn/ members/ hhan

Spatio-Temporal Statistics with R Chapter Two: Exploring Spatio-Temporal Data Spatio-Temporal

Spatial and temporal Spatial and temporal changes in Namaqualand Namaqualand: : changes in

Automated Reasoning in Modal, Tense and Temporal Logics Rajeev Gor c e Logic and

Introduction to Modal and Temporal Logic c Rajeev Gor e Automated Reasoning Group

INF@AVS 2018: Learning discrete and continuous representations for cross-modal retrieval

Directions in Dialogue Research (Engineering Applications in mind) Speech only Multi

Task adjustment options: You may select one of the poems provided by your teacher, however, please

Water Resources Water Resources Water Resources Water Resources Geospatial World Forum 2014

Modal and temporal logic N. Bezhanishvili I. Hodkinson C. Kupke Imperial College London 1 / 83

Efficient representations for the modal logic S5 Alexandre Niveau Bruno Zanuttini GREYC lab,

Outlines Effects of Lexical Spatial-Temporal Metaphor l Introduction l Experiment 1: Survey on

Investigating the forensic applications of global and local temporal representations of speech for

P. falciparum: Examination of Correlation Between Spatial Location and Temporal Expression of