Outline Human-Centered Perspectives in � Introduction Image Retrieval � Related Work � Levels of Description � Types of Users Alex Jaimes � Types of Search and Image Uses Oct. 9, 2007 � Personal Factors IDIAP Research Institute, Martigny, � Conclusions & Future Work Switzerland The Media Revolution [A non-mathematical historical perspective] What is happening? � Multi-cultural, multi-lingual environments, large (and instant) Future access-to and storage-of multimedia information (documents, Super 8 mm Applications sensors: RFID, etc.) Film 2050+ Lithography Cartridges VCR TIVO 1798 1965 1972 TV Anytime � A variety of devices (cell phones, meeting rooms, desktop Photography Late 1990s 1860s systems) and media (voice, video, text) for access, different band- User Digital widths Activity Brownie camera Cameras 1900 1990s YouTube � Differences across time and space, lower communication costs, Flickr more asynchronous collaboration, annotated collections (communities and social networks). Time Interactive Media! Human-Centered? What Is a Human-centered System? � A system that involves any human activity – Multimedia indexing (humans use images and video) – Camera-based Human-computer Interaction – Understanding of any sensory perceivable actions (e.g., eye, any body-part movement, emotions) and whose design uses human models or gives special consideration to human abilities – Utilize human memory, subjectivity, etc.
Image Retrieval Related Work � Why image retrieval? � Technical Advisory Service for Images (http://www.tasi.ac.uk/) – Personal use • Search vs. browse � Belkin et. Al. (Microsoft) [SIG CHI ’05] • Organize vs. create � Brajnik et. al. (U. Udine) [SIGIR ’96] � Pisciotta et. al. (Penn State) User Study [’01- � Why human factors? ’05] – Most images record human activities and are � Christel et. al. (CMU) [CIVR ’05] used for human activities � Hollink et. al. (U. Amsterdam) [ Intl. J. of Human Comp. Studies ’04] Feeling? Or Levels of Description & Meaning Example: “blue” Color? � Images can be described at multiple levels – Syntax { local, global } – Semantics { of, about } � Meaning of images is emergent (Santini et. al.) – Collection specific – Task specific Context – Person specific – Time specific Type? Action? Of? Object? Example: “painting” Example: “George Bush” What kind? About?
A white house Or Example: “white house” Time & Context “the” White House? Peaceful, Time & Context Time & Context Calm.. ? Time & Context Depressing! Levels & Meaning � What is the problem? – Data can be indexed at multiple levels – System’s indexing level and user’s level do not match – Indexing is static. But meaning is dynamic (context changes!)
Examples Levels & Meaning � What are the solutions? – Index at multiple levels • Understand data, understand users, use context – Obtain context information from the user (which white house are you looking for? Picture of or about white house?) – … but what about dynamic semantics? Open issue! Multi-Level Indexing Pyramid Multi-Level Indexing Pyramid � Key ideas � Conceptual structure for classifying visual attributes into multiple levels – Of vs. About – Art (E. Panofsky), cognitive psychology (E. Rosch et al.) – Information sciences (C. Jörgensen), visual information retrieval – Syntax vs. Semantics � Why the pyramid? – Percept vs. Concept – Represents full range of visual attributes – Semantic vs. Affective – Strong impact on MPEG-7 – Can also be used for audio, and video Multi-Level Indexing Pyramid Multi-Level Indexing Pyramid � Key ideas � Conceptual structure for classifying visual attributes into multiple levels – Of vs. About – Art (E. Panofsky), cognitive psychology (E. Rosch et al.) – Information sciences (C. Jörgensen), visual information retrieval – Syntax vs. Semantics � Why the pyramid? – Percept vs. Concept – Represents full range of visual attributes – Semantic vs. Affective – Strong impact on MPEG-7 – Can also be used for audio, and video
Indexing Levels (Visual Attributes) Level 1: Type/technique Knowledge Syntax 1. � Type/technique used 1. Type/ Syntax during production Type/ Technique 2. Technique Global Distribution 2. 3. Global Distribution Local Structure Texture, etc. � No knowledge of visual 3. 4. Local Structure Global Composition 4. content, just general Global Composition 5. visual characteristics Generic Object 6. Semantics Generic Scene 7. Specific Object � Examples: 8. Specific Scene 9. – Color or b/w Abstract Object 10. photograph Ana Alex Abstract Scene Oil painting – Water color, oil B/W photograph painting, mixed media Level 2: Global Distribution Level 3: Local Structure � Characterization and 1. � Distribution of low-level Type/ Syntax extraction of basic 1. Technique features only Type/ 2. Syntax visual elements Global Distribution Technique 3. 2. Local Structure Global Distribution 4. � Examples: 3. Global Composition Local Structure � Examples: – Color distribution 4. Global Composition • Dominant, histogram – Dots, lines, tone, circles, – Global texture squares • Coarseness, contrast – Local color – Global shape – Binary shape mask • Aspect ratio – Local motion/deformation – Global motion/deformation Similar texture, color histogram • Speed, acceleration Blood cells = circles Stars = dots Level 4: Global Composition Level 5: Generic Object � Arrangement or layout of 5. Generic Object 1. 6. basic elements � General (every day) Type/ Generic Scene Syntax Semantics 7. Specific Object knowledge about Technique � No knowledge of objects 8. 2. Specific Scene Global Distribution objects 3. 9. Local Structure Abstract Object � Examples: 10. 4. Abstract Scene Global Composition – Balance, Symmetry � Examples: – Center of interest – Leading line, viewing angle – Common nouns • Person • Chair • Desk Airplane Persons, flag What the image is “of” Horizontal leading line Centered object Centered object
Level 6: Generic Scene Level 7: Specific Object � Identified and named 5. 5. Semantics Generic Object Semantics Generic Object objects � General knowledge 6. 6. Generic Scene Generic Scene 7. 7. about scene Specific Object Specific Object � Specific knowledge 8. 8. Specific Scene Specific Scene 9. 9. about objects, known Abstract Object Abstract Object 10. 10. Abstract Scene facts Abstract Scene � Examples: � Examples: – B. Clinton – City, Landscape – Chinese Ambassador Z. Li – Indoor, Outdoor – American flag – Daytime, Nighttime – Lincoln desk Outdoors, city, street Indoors, office F-18 B. Clinton, Z. Li What the image is “of” What the image is “of” Level 8: Specific Scene Level 9: Abstract Object 5. � Interpretation of an � Identified and named Semantics Generic Object 5. 6. object Generic Object scene Semantics Generic Scene 6. 7. Generic Scene Specific Object 7. 8. Specific Object Specific Scene 8. 9. � Specific knowledge Specific Scene � Subjective or based on Abstract Object 9. 10. Abstract Object about scene, known Abstract Scene specific personal 10. Abstract Scene facts knowledge � Examples: � Examples: – Name of a city, street, lake – Political power – Name of a building – Sympathy Political Gesture What the image is “of” About music? or trial? What the image is “about” Paris Oval Office, White House Level 10: Abstract Scene Pyramid Example 1. TYPE: Color still image 5. Syntax Generic Object 2. GLOBAL DISTRIBUTION: Color histogram Semantics 6. � Subjective Generic Scene Circles, squares 3. LOCAL STRUCTURE: 7. Specific Object interpretation of a 8. 4. GLOBAL COMPOSITION: Centered Specific Scene 9. Abstract Object scene 5. GENERIC OBJECT (of) : Persons, building 10. Semantics Abstract Scene 6. GENERIC SCENE (of) : Outdoors 7. SPECIFIC OBJECT (of) : Ana, Alex 1. � Examples: Type/ 8. SPECIFIC SCENE (of) : CEPSR Technique 2. 9. ABSTRACT OBJECT (about) : Happy, friendly Global Distribution – International politics 3. Local Structure 10. ABSTRACT SCENE (about) : Research agreement, 4. Global Composition – War 5. friendship Generic Object 6. Generic Scene – Apology 7. Specific Object Peacefulness US Government 8. Specific Scene 9. What the image is “about” Abstract Object 10. Abstract Scene
Recommend
More recommend
