On Quantizing the Mental Image of Concepts for Visual Semantic - PowerPoint PPT Presentation

On Quantizing the Mental Image of Concepts for Visual Semantic Analyses Marc A. Kastner (Nagoya University) Doctoral Symposium #3 Supervisors: Dr. Ichiro Ide, Prof. Hiroshi Murase

Visual variety How broad is a term? High? Low? Lamborghini Object Ground vehicle • Vehicle • Sports car Car Aventador 1.3 3 5 ・・・ • Different backgrounds • Same form 6.5 Vehicle Sports car Motor vehicle • Different forms/colors • Backgrounds are similar 2.2 5.8 4.5  Low value  High value Concrete Abstract 2

Imageability of words • Concept from Psycholinguistics [1] • Quantize the perception of words • Often described on Likert scales • Unimageable ⬌ Imageable, or Abstract ⬌ Concrete • Is a concept imageable? Do you have a mental image when thinking of a concept? Car Ca Vehic icle le So Somethin ing Peaceful (6.7) (1.6) (3.4) (5.5) Imageable (Concrete) Unimageable (Abstract) 3 1: Pavio et al. Concreteness, imagery, and meaningfulness values for 925 nouns. J Exp Psych 1968.

Core ideas • Estimate the mental image of things for multimedia modelling • Imagine different concepts • Are they hard to visually imagine? • Are they rather abstract or concrete? • Goals • Use images from social media and the Web to estimate mental image of things • Evaluate the semantic gap between concepts by first quantizing it 4

Research 1: Dataset-driven Pictures of: Sports car Pictures of: Jeep • Create less biased ImageNet & datasets Web-crawling … Google Image • Re-composite datasets by Search #results sports car 27.4% racer 9.2% using ratio of sub- Model T 8.8% coupe 6.9% concept popularities used-car 6.7% jeep 5.0% beach w. 4.8% compact 4.5% • E.g. vehicle consists of: cab 3.9% convertible 3.5% hatchback 2.7% Re-composited dataset for car many cars , few tanks minivan 1.3% ambulance 1.4% 5

Research 2: Algorithm- Input: driven Images for “leaf” For each visual feature 𝑔 𝑗 • Use visual data mining on Feature vector for 𝑔 𝕨 𝑗 𝑗 crawled images • YFCC100M Cross comparison between all images for “leaf” 1.0 0.3 ⋯ • Use combination of low- Similarity matrix 𝑡 𝑗 = ⋮ ⋱ ⋮ 0.7 ⋯ 1.0 and high-level features Train on eigenvalues Imageability dictionary Random Regressor Forest • Train using psycholinguistic Output: dictionary as ground-truth Imageability for “leaf” 𝐽 leaf ∈ [1,7] 6

Thank you for your attention! Questions? kastnerm@murase.is.i.nagoya-u.ac.jp https://www.marc-kastner.com/ @mkasu 7