20/09/2018 Media Indexing & Retrieval Media Indexing & Retrieval Prepared by Ling Guan Jose Lay Paisarn Muneesawang Ning Zhang Rui Zhang 1 Background • More and more media are becoming available online. In the past decade, we have seen the proliferation of online newspaper, net radio, web ‐ TV, net games, etc. etc. • The size of the Internet is estimated to have exceeded 13,706,770 PB (Petabytes = 1,000 TB), while the worldwide disk storage capacity in 2018 is in the volume of 1450 EB (Exabyte = 1,000,000 TB). • Newly created digital data in the year of 2011 is about 1800 Exabyte. • Over 60% of the online data are of audio and/or visual content. • Naturally the information contained in these massive data would need to be accessible for them to be useful. • Thus, Multimedia Information Retrieval 2 1
20/09/2018 Prominent Venues for Indexing and Retrieval • IEEE International Conference on Multimedia and Expo (ICME). • ACM International Conference on Multimedia (ACM ‐ MM). • IEEE Symposium on Multimedia (ISM). • ACM International Conference Multimedia Information Retrieval (ACM ‐ MIR). • IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR), inaugurating in 2018. • The list is long… 3 Interesting Facts about ICME • ICME 2005: 380 out of the 850 papers submitted are related to media indexing and retrieval. • ICME 2006 (Organized by Ryerson University in Toronto): about 500 out the 1200 papers submitted are related to media indexing and retrieval. • ICME 2005: 80% of the industrial exhibits are on digital asset management, a principal application area of multimedia retrieval. • ICME 2009, 30% of the papers are on multimedia indexing and retrieval. • ICME 2014, 40% of the papers are on indexing and retrieval, and digital libraries. 4 2
20/09/2018 Objectives • To understand multimedia information retrieval from: – the intellectual foundation of the subject. – the operational techniques of the processes. 5 Outlines • Introduction: – Intellectual Foundation of Multimedia Information Retrieval – Retrieval Models • Text Retrieval: – Database, Bibliographic, and Keyword Searches • Content ‐ based Retrieval: – Object ‐ matching and beyond • Indexing: – Inverted file • Searching: – Multimodality and Query adaptation 6 3
20/09/2018 Intellectual Foundation • Multimedia, Information, and Retrieval – Multimedia characteristics. – The difference between Data and Information. – Data Retrieval and Information Retrieval. – The traditional information retrieval (IR) objectives. • Multimedia Information Retrieval • Multimedia Information Retrieval Objectives 7 Multimedia • Multiple media sources • Multimodality • Coordination • Interactivity 8 4
20/09/2018 Data and Information Traditional bibliographical organization differentiates the notion of work as opposed to document. • Work is: – an intellectual creation. – the disembodied message, the information. • Document is: – the material embodiment of an intellectual creation, the data. 9 Data and Information Retrievals • Information (or work) differs from data (or document). • Information Retrieval should not be confused with Data Retrieval. 10 5
20/09/2018 Retrieval A duo of Indexing and Searching primitives. • Indexing (annotation) deals with: – Finding representations for information in the document. – Organizing the representation to facilitate efficient search. • Searching deals with: – Capturing and presenting an information need. – Assessing relevance. 11 Traditional IR Objectives • To enable finding a document of which: – The title, the author, or the subject is known. • To locate a collection of documents: – By an author, a subject, a specific kind of literature. • To facilitate the choice of a document based on: – Its edition or literary association. 12 6
20/09/2018 Multimedia Information Retrieval Multimedia IR can be presented in many ways: • Extension View • Functional View • Modality View • Topical View • Other Views 13 Extension View • Traditional Information Retrieval techniques, e.g. bag of words, TD ‐ IDF, can be extended to deal with various data types as a document could take any form. • Operationally it is being challenged by content ‐ based techniques. 14 7
20/09/2018 Functional View • Indexing: INDEXING – Information Description Information Information – Descriptions Organization Identification Description • Searching: Documents – Query Description DB – Relevance Assessment Practically, information is Information Query reduced to mere viewpoints Identification Description (e.g. car show with a lady in SEARCHING Information Needs the front). 15 Modality View • Number of modalities – Single Modality Audio – Multiple Modalities • Chronological Image MM Video – Text ‐ based – Content ‐ based – Concept ‐ based (semantic) – Context ‐ based Text 16 8
20/09/2018 Modality View (2) Multimedia Document Text Audio Video Keywords Segment Region Global Meta Structure Temporal Semantic Spatial • Multimedia Description Semantics 17 Topical View Multimedia Information Retrieval Storage Technology and Library and Distributed Processing Information Science Data Structure and Pattern Recognition and Algorithms Computer Vision Psychology and Artificial Intelligence Human Computer Interaction & Machin Learning 18 9
20/09/2018 Other Views • Multimedia IR could also be looked at from many other perspectives: – Push and Pull Directives – Structured and Unstructured Data – Real ‐ time and Offline Processing – Automatic and Manual Indexing – Compressed and Spatial Domain Descriptions – And many more. 19 Multimedia Information Retrieval Objectives Traditional Bibliographic Objectives Object Matching Keyword Searches 20 10
20/09/2018 Flag Pole End of Introduction. 21 Text Retrieval Text retrieval could normally be classified into two categories: • Structured – Database Structured Unstructured – Bibliographic Systems • Unstructured – Full ‐ text or Keyword Search 22 11
20/09/2018 Database Systems • In a database system: – relations and other attributes are formally constructed. – Search is limited by those attributes and their relations. 23 Bibliographic System • Click the screen above to start the demo 24 12
20/09/2018 Bibliographic Objectives Restated • To enable finding a document of which: – The title, the author, or the subject is known. • To locate a collection of documents: – By an author, a subject, a specific kind of literature. • To facilitate the choice of a document based on: – Its edition or literary association. 25 Bibliographic Operation • Documents are indexed by using surrogates. • A surrogate is a description of an information viewpoint. • Document databases are reduced to surrogate databases. • Search are performed on the surrogate databases. 26 13
20/09/2018 Bibliographic Constraints • The manual annotation for describing any information viewpoint is a costly and slow process. • Clearly it stands as the major handicap. To reduce the cost and speed up the process: – Standardization. Thus, interoperability. Distributing cost among institutions. – Automatic Indexing. 27 Full ‐ Text Retrieval • Full ‐ text retrieval is the champion of the automatic indexing efforts. • It compromises the advanced features of bibliographic system such as collocation with speed and significantly cheaper cost. 28 14
20/09/2018 Full ‐ text Retrieval Click on the screen to start the demo. 29 Full ‐ text Retrieval: another example 30 15
20/09/2018 Full ‐ text Retrieval Principle • Luhn’s Ideas: – The frequency of word appearance in an article furnishes a useful measurement of word significance. – Words exceeding upper cut ‐ off are too common, while below lower cut ‐ off are rare. Significant words lie in between. 31 Full ‐ text Retrieval Characteristics • makes no attempt to understand “information” (work) by dealing only with data such as using the statistical measures. • is essentially a “data retrieval” scheme. • has been highly successful and forms the back bone of the Web Search Engine. 32 16
20/09/2018 Full ‐ Text Indexing • Inverted File Indexing: – Consider a document: Content Retrieval using Energy Histogram with LF ‐ DCT Coefficient and Segment Grid. – A list of keywords between lower and upper cut ‐ offs are extracted: Content, Retrieval, Energy, Histogram, LF ‐ DCT, Coefficient, Segment, Grid. – The keywords are used to populate the inverted index file. … 33 Full ‐ Text Indexing In an inverted index for 20 documents: …. • Coefficient = {1, 4, 18} • Histogram = {1, 20} … • Retrieval = {1, 3, 7, 9, 12, 15, 18, 20} … The numbers within a bracket refers to the document(s) in which a particular keyword is found. 34 17
Recommend
More recommend
