High Level Semantic Modeling Shih Fu Chang Digital Video Multimedia - PowerPoint PPT Presentation

High ‐ Level Semantic Modeling Shih ‐ Fu Chang Digital Video Multimedia Lab, Columbia University CVPR Tutorial, June 2014 1200 SentiBank Predict Sentiment Concepts

Interestingness Aesthetics Beyond Semantics Style Emotion Others:, Creativity, Intent, Memorable …

Difficult Problems ‐ but interesting datasets & results emerging

Visual Aesthetics • Datta et al ECCV 2006, Naila Murrary et al CVPR 2012 (AVA) • AVA Dataset: 250,000 images from 963 dp ‐ challenges with aesthetics scores and semantic/style labels

Aesthetics is Subjective ‐ Non ‐ Conventional Style/Subject Tends to Cause Large Score Variations Murrary et al CVPR 2012 (AVA)

Score Distributions of Each Image Vary but Form Patterns Murrary et al CVPR 2012 (AVA) Gaussian distribution a reasonable fit

Semantics Also Plays An Important Role ‐ Many less attractive classes are associated with negative semantics Murrary et al CVPR 2012

Despite these… Big Dataset/Complex Model Help Fisher Vector #Components Murrary et al CVPR 2012

What makes video pleasing – NHK 1000 Videos Aesthetics Ranking at ACMMM13 Aesthetically Pleasing Not ‐ so Pleasing Cinematographic Additional Filter for Personal Video Evaluation Web video search Collection (Bhattacharya et al, ACMMM 2013) 9

Computational Video Aesthetics (Bhattacharya et al, ACMMM 2013) (Subh Bhattacharya et al, ACMMM 2013)  Camera motion Shot Level  Foreground Aesthetic Motion Query Models  Texture Dynamics Video Shot Fused Frame Level  Input Video Semantics Aesthetic Aesthetic  Sentiments Models Model Keyframe Predicte  Sharpness Cell Level Appeal  Eye Sensitivity Aesthetic  Dark Channel Models Cell

Predicted Ranking of Video Aesthetics (Bhattacharya et al, ACMMM 2013)

Modeling Interestingness (Flickr Explore Rank Order) Sagnik Dhar et al, 2011 S.F. Chang 12

Which Images are More Memorable (Philip Isola et al, 2011) ‐ sky, tree, mountain + person, floor, car S.F. Chang 13

Interestingness vs. Memorability vs. Aesthetics Michael Gygli et al, 2013 S.F. Chang 14

For Content to be Viral, it Needs to be Emotional ‐ Dan Jones Plenty on the Web: Psychology emotion wheel “ For content to go viral, it needs to (8 emotions, by Robert Plutchik) be emotional,” Dan Jones 15

The Power of Social Visual Multimedia 2012 Tweets of the Year @BarackObama: Four more years. @Brynn4NY: Rollercoaster at sea. 16

Classifying Image Emotions Machajdik and Hanbury, ACMMM 2010 IAPS Affect Data set Art Affect Data set 17

How Do People Describe Emotions in Web Photos? ‐‐ Web mining to discover visual emotions in social media MISTY Build WOODS Sentiment Ontology Select Concepts Discover sentiment Psychology emotion words SAD EYES wheel (8 emotions) Robert Plutchik, ‘91 Analyze tags with strong sentiments Borth, Ji, Chen, Breuel, Chang, Large ‐ Scale Visual Sentiment Ontology , ACM Multimedia 2013 18

Concurrent tags with emotions From 6 million tags on Flickr and YouTube Color code: text sentiment values S.F. Chang 19

Frequent Photo Tags Related to Emotions S. ‐ F. Chang

From Machine Vision Perspective: Not all concepts/entities are detectable! ‐‐ which 1000 concepts to focus in pictures?

Computational Focus – Adjective ‐ Noun Pair (ANP) • Adjective (268): express emotions • positive: beautiful, amazing, cute • negative: sad, angry, dark • Nouns (1187): possible detection • people, places, animals, food, objects, weather • Standard steps: – remove entities like “hot dog” via wikipedia – Choose sentiment rich ANP concepts by NLP tools “Senti ‐ WordNet” “SentiStrength” S.F. Chang 22

ANP Ontology (noun) • 6 levels – ANIMALS – FLORA – PERSON – OBJECTS – NATURAL PLACES – MAN ‐ MADE PLACES – VEHICLE – FICTIONAL_CREATURES – FOOD – ABSTRACT_CONCEPTS – ART_PHOTOGRAPHY – EVENTS – ACTION – WEATHER_CONDITIONS – TIME S.F. Chang 23

ANP Ontology (adjective) • adjectives (2 levels) – weather (stormy, cold, sunny) – people (young, attractive) – animals (cute, fluffy) – places (haunted, misty) – food (yummy, salty) – object (colorful, beautiful) S.F. Chang 24

Visual Sentiment Ontology (Browser)

Visual Sentiment Ontology (Browser)

Open Issues … • How will the visual sentiment ontology change over different domains? – Differences in photo style, quality, user groups, culture, tasks • How to link mid ‐ level concepts to high ‐ level emotions? – currently based on association

Next Step: Teach Machine to Recognize Visual Sentiments Build Sentiment Ontology MISTY WOODS Train Classifiers Select Adj ‐ Noun Pairs Discover sentiment Performance Psychology emotion words SAD EYES Filtering wheel (24 emotions) SentiBank (1200 Detectors) Sentiment Prediction S.F. Chang 28

Image Features • Generic features – Color Histogram (3x256 dim.) – GIST descriptor (512 dim.) – Local Binary Pattern (52 dim.) – SIFT Bag ‐ of ‐ Words (1,000 codewords 2 ‐ layer spatial pyramid, max pooling) – Attribute descriptor (2,000 dim.) • Special features – Object detection (people, objects, etc.) – Aesthetics features (color schemes, layout, etc.) – Face and attributes – Improve accuracy 9% ‐ 30% S.F. Chang 29

Aes Aestheti hetic Fe Feat atures • Dark Channel [He et al. ‘09] • minimum of local intensity • Sharpness [Vu et al. ‘11] • sharpness of local image regions by spectral and spatial measures • Depth of Field • wavelet decomposition in HSV color space (low vs high) • Color Harmony [Nishiyama et al. ‘11] • Using local histogram of Moon ‐ Spencer model, which defines compatibility of two color values (example of compatible color )

Wh What at Do Do Hum Humans ans Expect Expect to to See? See? ‐ from small annotation experiments Jules3000@flickr HIKARU Pan@flickr houseofduke@flickr masochismtango@flickr “Smiling dog”: tongue visible, mouth open, face camera, close shot, pink tongue, open mouth, frontal dog face houseofduke@flickr hurlham@flickr ebonique2007@flickr springlakecake@flickr “Tired dog”: lying on floor or surface, closed eyes, yawning, resting, no action, fore legs, paws, face on floor

So So, Need Need to to Link Link to to Obj Object cts + At Attributes Bahman Farzad@flickr SamFan1@flickr Training Images of Same Noun (e.g. dog) Object (noun) Detector HOG, DPM zoompict@flickr Yes No Feature Extraction Discard BloodyGoku21@flickr • object/background/whole Soft Adj. Labels • SIFT, GIST, LBP, color ConceptNet, • aesthetics (symmetry, SentiStrenth, white balance, etc.) human labels • composition (object size, position) Adjectives: Weighted SVMs: cute sad wet ANP Classifiers + cute sad wet dog dog dog Feature selection:

Hier Hi erar archi chical al: Obj Object ct + Af Affect ct Attri tribut butes es Testing Images • Testing dog? face? car? Candidates + Features Candidates + Features Candidates + Features ANP Classifiers: sweet cute sad wet mad silly hot tiny safe dog? dog? dog? face? face? face? car? car? car? Fuse Noun Score: Max Score Output: Karf Oohlu@flickr rollinoldman@flickr NiH@google+ epSos.de@flickr green_lover@flickr paevalill@flickr houseofduke@flickr ccdoh1@flickr flatworldsedge@flickr

Tricky Issue: Concept Subjectivity • Attribute s subjective, ambiguous, and overlapped – E.g., cute dog, fluffy dog, cuddly dog • Solution – Need a way to handle soft label overlap – Model overlap proportion in Cute dog SVM Fluffy dog Tiny dog

The S VM Algorithm F. Yu; D. Liu; S. Kumar; T. Jebara; S. ‐ F. Chang. ∝ SVM for learning with label proportions. ICML13 Label prediction loss proportion loss • Learned with alternate optimization or a relaxed convex form • Formulation: Image set of “Fluffy Dog” has proportion p k being “Cut Dog” Image set of “Tiny Dog” has proportion p k being “Cut Dog” proportions can be approximated by ConceptNet 35

Example: S VM for Video Event Recognition K. ‐ T. Lai; F. X. Yu; M. ‐ S. Chen; S. ‐ F. Chang. CVPR 2014 • Model the proportion of positive instances in each event • Detecting complex events in ~ 100,000 videos with 20% gain

Sen SentiBank iBank 2: 2: Obj Objects w. w. Attri tribut butes es SentiBank 1 SentiBanks with Object/Attribute

Detection Examples S.F. Chang 38

VSO/SentiBank Resources Ontology and 1,200 Classifiers http://visual ‐ sentiment ‐ ontology.appspot.com/ Shih ‐ Fu Chang 39

Application: Live Sentiment Prediction 1200 Predict Sentiment Classifiers PhotoTweet Stream: Positive? Neutral? Negative? True stuff. I have mad respect for all the True stuff. I have mad respect for all the ladies that DO NOT give in to abortion. ladies that DO NOT give in to abortion. #groundzero #hurricanesandy #groundzero #hurricanesandy @nickespo89 #newjersey #newjersey Ouch mr police man @radiodario @charleslawrence 40