Deep Learning Based Semantic Video Indexing and Retrieval Anna - PowerPoint PPT Presentation

Deep Learning Based Semantic Video Indexing and Retrieval Anna Podlesnaya, Sergey Podlesnyy Cinema and Photo Research Institute (NIKFI) This work was funded by Russian Federation Ministry of Culture Contract No. 2214-01-41/06-15

Fast Track Contribution 1 Contribution 2 Contribution 3 Video Segmentation Video Indexing Search by Examples Feature vector extracted Graph-based database Video retrieval by sample by GoogLeNet contains for temporal, spatial and video clip @0.86 enough semantic semantic properties precision information for indexing is proposed. segmenting raw video Online learning of new into shots with 0.94 Cost efficient pipeline. concepts: video retrieval precision compared to by sample photos @0.64 MPEG-4 i-frames. precision.

Relevance Archives are Huge Production Needs MPEG-7 Query Format Russian Documentary Everyday need for QueryByFreeText ● Archive: 250K items footage in TV production QueryByMedia ● (dated from 1910) ● SpatialQuery Non-fiction movies ● TemporalQuery Russian TV Archive: production relies on 100K items historical and cultural heritage content ISO/IEC 15938-5:2003 Information Youtube: users uploading technology -- Multimedia content 100hrs of video every Education, research, art... description interface -- Part 5: minute (as of 2013) Multimedia description schemes

Semantic features extraction by Video deep neural network Segmentation Shots cut by vector distance spikes between frames With semantic features Temporal pooling for shot semantics summarizing

Deep Neural Network

Semantics Feature Vector

Distance Between Frames’ Feature Vectors

Segmenting Algorithm Details { x 0 , x 1 , … x n } — feature vectors of successive frames

Robustness to Camera Movement Zoom Pan/Rotate Pan Pan Zoom/Pan

Apache Cassandra storage for feature vectors and thumbnails Video Indexing Neo4j graph database for movies archive With graph database Structured queries for keywords-based retrieval

Starting with film or Store per-scene data Add edges to Neo4j tape structure in Neo4j graph to speed up nearest neighbors search FV BK-Tree Digitizing Segmentation Indexing Extraction Building Store per-frame May use additional timecodes and feature classifiers for faces, vectors in Cassandra places, salient objects etc.

Graph-Based Index

Neo4j Graph

Neo4j Graph

Neo4j Query Find Scenes with Zebra MATCH (s:Shot) - [c:Category] -> (w:Wordnet {synset: “zebra”}) WHERE c.weight > 0.1 RETURN s ORDER BY s.duration DESC ASCII art: (s)-[c]->(w)

Neo4j Query Find Scenes with Lion at Left to Zebra MATCH (s:Shot) --> (zebra_obj:Salient_obj) --> (w:Wordnet {synset: “zebra”}) MATCH (s) --> (lion_obj:Salient_obj) --> (w:Wordnet {synset: “lion”}) MATCH (zebra_obj) - [:Left] -> (lion_obj) RETURN s ORDER BY s.duration DESC (s)-->(zebra_obj)-->(w) (s)-->(lion_obj)-->(w) (zebra_obj)-[:Left]->(lion_obj)

Search by Find similar clip Example Find near-duplicates Online learning of new concepts One picture is better than 100 words

Use Case 1 Found clips with required characteristics Search for ELEPHANT Keyword Search Select Sample Clip Find Similar Clips Need elephants herd, forest, sky

Find Similar Clip Quick Search Exhaustive Search An average precision of search by video sample was 0.86. The 31-bit random projection Feature vectors pooled precision was evaluated by by scene ( R 1024 ) hash (RPH) searching by a keyword and then searching by one of resulted shots with cosine BK-Tree on PRH Cosine distance between distance threshold 0.3. A hamming distance from sample clip and every human expert performed sample clip other scene in the true/false positives counting. archive, sort descending Quick incomplete search Well, slow

Found near-duplicates, Use Case 2 robust to resampling, vignetting, hue/sat Show sample clip augmentation etc. Extract Features Exhaustive Search Sort by Cos-distance 0.0057 0.0124 0.0152 0.0583

Found video clips Use Case 3 matching the classifier trained on images Show sample images feature vectors of unknown concept AP 0.64 Google Image Search Train Linear Classifier Exhaustive Search Vowpal Wabbit

Future Work Thank you! ● Faces ● Places Questions welcome Video to text annotations ●