Real-Time Computerized Annotation of Pictures Jia Li James Z. Wang - PowerPoint PPT Presentation

Real-Time Computerized Annotation of Pictures Real-Time Computerized Annotation of Pictures Jia Li James Z. Wang The Pennsylvania State University Email: jiali@psu.edu, jwang@ist.psu.edu Jia Li, James Z. Wang alipr.com

Real-Time Computerized Annotation of Pictures How Visible Are Web Images? Keukenhof photos Jia Li, James Z. Wang alipr.com

Real-Time Computerized Annotation of Pictures ALIPR: Automatic Linguistic Indexing for Pictures - Real Time plant, flower, tree, plant, landscape, people, people, tulip water, garden flower, animal, plant,lake, people, rural, wild-life, dog, building landscape Jia Li, James Z. Wang alipr.com

Real-Time Computerized Annotation of Pictures Architecture for Training Jia Li, James Z. Wang alipr.com

Real-Time Computerized Annotation of Pictures Image “Knowledge Base” Jia Li, James Z. Wang alipr.com

Real-Time Computerized Annotation of Pictures Six Hundred Semantic Categories ◮ Corel image database ◮ 80 images per category. ◮ Each category is described by several words: ‘‘autumn, tree, landscape, lake’’ . ◮ A total of 332 distinct words. Jia Li, James Z. Wang alipr.com

Real-Time Computerized Annotation of Pictures Feature Extraction ◮ Color components: LUV ◮ Texture features: wavelet coefficients Jia Li, James Z. Wang alipr.com

Real-Time Computerized Annotation of Pictures Region Segmentation and Signature Formulation Jia Li, James Z. Wang alipr.com

Real-Time Computerized Annotation of Pictures Region Segmentation and Signature Formulation ◮ An image signature resides in Ω = Ω 1 × Ω 2 . ◮ Color distribution: β i , 1 ∈ Ω 1 . ◮ Texture distribution: β i , 2 ∈ Ω 2 . ◮ β i , j = i , j ) , ..., ( v ( m i , j ) , p ( m i , j ) { ( v (1) i , j , p (1) ) } . i , j i , j Jia Li, James Z. Wang alipr.com

Real-Time Computerized Annotation of Pictures Profiling Image Concepts via Mixture Models Jia Li, James Z. Wang alipr.com

Real-Time Computerized Annotation of Pictures Mixture Modeling via Local Mapping ◮ Mixture modeling for space Ω ◮ Carve Ω into cells by clustering. ◮ Map each cell to an Euclidean space, preserving pairwise distances. ◮ Model the mapped points by Gaussian. ◮ Images: a grid of feature vectors ◮ Gaussian mixture ◮ 2-D HMM Jia Li, James Z. Wang alipr.com

Real-Time Computerized Annotation of Pictures Architecture for Training Jia Li, James Z. Wang alipr.com

Real-Time Computerized Annotation of Pictures Architecture for Annotation Jia Li, James Z. Wang alipr.com

Real-Time Computerized Annotation of Pictures Word Probabilities ◮ Total word list: Category prob. given signature β W = { w 1 , w 2 , ..., w K } . ρ m f ( β | M m ) p m ( β ) = ◮ Semantic categories containing � M l =1 ρ l f ( β | M l ) word w i : C ( w i ). ◮ Model of category m : M m , m = 1 , ..., M . Word probability ◮ Prior on categories: ρ m (set � q ( β, w i ) = p m ( β ) . uniform). m : m ∈C ( w i ) Jia Li, James Z. Wang alipr.com

Real-Time Computerized Annotation of Pictures Human Evaluation on flickr.com Images ◮ Manual evaluation on 5 , 411 flickr.com images. ◮ Accuracy of the first word: 51 . 17%. Jia Li, James Z. Wang alipr.com

Real-Time Computerized Annotation of Pictures Human Evaluation on flickr.com Images ◮ Coverage rate: percentage of images correctly annotated by at least one word. ◮ Top 4 words: > 80%. ◮ Top 7 words: 91 . 37%. ◮ Top 15 words: 98 . 13%. Jia Li, James Z. Wang alipr.com

Real-Time Computerized Annotation of Pictures Human Evaluation on flickr.com Images ◮ Annotate using top 15 words. ◮ # correct: 4.1 on average Jia Li, James Z. Wang alipr.com

Real-Time Computerized Annotation of Pictures Speed ◮ Training: ◮ 109 seconds on ave. ◮ 80 images per category ◮ 2.4 GHz AMD processor ◮ Annotation: ◮ 1.4 seconds on ave. for example images ◮ 3.0 GHz Intel processor ◮ Convert from JPEG to raw format; extract image signature; find annotation words. Jia Li, James Z. Wang alipr.com

Real-Time Computerized Annotation of Pictures Conclusions Learning methodology System ◮ D2-clustering ◮ The ALIPR system: real-time automatic annotation of ◮ Generalized k-means to pictures bags of weighted vectors ◮ Human evaluation on web ◮ Mixture modeling via mapping images to conjectural space ◮ Human evaluation on 5 , 400+ Web images has demonstrated promising results. ◮ Future work: bridge with retrieval, incremental learning, improve modeling, Web applications ... ◮ ALIPR your pictures: http://alipr.com Jia Li, James Z. Wang alipr.com

Real-Time Computerized Annotation of Pictures Jia Li, James Z. Wang alipr.com