Robust Scene Categorization by Learning Image Statistics in Context for BBC rushes Jan van Gemert,Jan-Mark Geusebroek, Cees Snoek, Dennis Koelma, Cor Veenman, Frank Seinstra, Marcel Worring and Arnold Smeulders Intelligent Sensory Information Systems University of Amsterdam
Overview • Visual Concepts by Scene Recognition Introduction Visual Features • How to Sky Sky Sky Sky Contextures recognize an Evaluation Sky Sky Sky Conclusions airplane? Sky Sky Context! • Use Proto-Concepts to Describe Context Slide 2 Robust Scene • SVM: link Context to Concepts Categorization by Learning Image • Learn Models on News data, evaluate on BBC rushes Statistics in Context
Low Level Features • Color Invariance Introduction Visual Features Contextures Luminance E λ Evaluation Conclusions E Blue-Yellow λ λ E Red-Green λλ λ Slide 3 Robust Scene • Invariant to Global Illumination Changes Categorization by Learning Image Statistics in Context
Natural Image Statistics • There are more non-edges than edges Introduction Visual Features Contextures Evaluation Conclusions • Distribution of Edge responses: Integrated Weibull Slide 4 � � γ γ − µ � � 1 r Robust Scene − exp 1 � � 1 γ β Categorization by 1 / γ � � γ Γ 2 ( ) Learning Image γ Statistics in Context
Proto-Concepts Building (321) Sky Car (192), Introduction Charts (52) Crowd (270) Visual Features Sky Desert (82) Contextures Fire (67) Flag_USA (98) Evaluation Flag_USA Maps (44) Conclusions Mountain (41) Road (143) Sky (291) Smoke (64) Snow (24), Vegetation (242) Sky Water (108) Slide 5 Robust Scene In brackets: nr. Annotations Categorization by Road at least 20 frames Learning Image Statistics in Context
Region Detection Introduction Visual Features Contextures Evaluation Conclusions Slide 6 Robust Scene Categorization by Learning Image Statistics in Context
Contextures • Contexture: Occurrence Histogram of Proto-Concepts Introduction Global (Accumulate) Local (Arg Max) Visual Features Contextures Evaluation Conclusions Building Building Building Building Building Building Building Building Building Building Building Car Car Car Car Car Car Car Car Car Car Car Road Road Road Road Road Road Road Road Road Road Road Sky Sky Sky Sky Sky Sky Sky Sky Sky Sky Sky … … … … … … … … … … … Building Car Road Sky … Slide 7 Robust Scene Categorization by Learning Image Statistics in Context
Learning Concepts in Video • Image to Shot: sample every second. Introduction Visual Features • Use SVM to link Contextures to 101 Concepts Contextures Evaluation • Performance on TrecVid Testset Conclusions Visual Only Best-Submission Average Performance 0.6 0.5 0.4 0.3 0.2 0.1 0 Slide 8 n n g A s r s g e r a o e n p i t n p S a n r i C i i a a o d s t k U o n M p c o l l u s a i _ s Robust Scene u l S p i g o W r r B e x M P a t E l a F W Categorization by Concepts Learning Image Statistics in Context
BBC Rushes • Evaluate the SVM-models trained on Introduction Visual Features TRECVID data on the BBC rushes Contextures • 25 ‘survive’: Evaluation Conclusions • aircraft, bird, boat, building, car, charts, cloud, crowd, face, female, food, government building, grass, meeting, mountain, outdoor, overlayed text, sky, smoke, tower, tree, urban, vegetation, vehicle, Slide 9 Robust Scene waterscape Categorization by Learning Image Statistics in Context
BBC rushes Screenshots (I) • Building Introduction Visual Features Contextures Evaluation Conclusions • Tower Slide 10 Robust Scene Categorization by Learning Image Statistics in Context
BBC rushes Screenshots (II) • Face Introduction Visual Features Contextures Evaluation Conclusions • Food Slide 11 Robust Scene Categorization by Learning Image Statistics in Context
Conclusions & Discussion • Does a picture say more than a thousand words? Introduction – According to our Trec results: Not (yet) Visual Features • Robust methods provide a rich untapped information source: Contextures – Re-use of annotations Evaluation Conclusions – Re-use of Training Models – Ideally: train a concept once, apply everywhere Slide 12 Robust Scene Categorization by Learning Image Statistics in Context
Recommend
More recommend
