EE 6882 Statistical Methods for Video Indexing and Analysis Fall 2004 Prof. Shih-Fu Chang http://www.ee.columbia.edu/~sfchang Lecture 1 - part B (9/7/04) 1
Run-through of a simple image search system Color, Texture, distance metrics, and evaluation issues � References � J. R. Smith and S.-F. Chang, "VisualSEEk: A Fully Automated Content-Based Image Query � System," ACM Multimedia Conference, Boston, MA, Nov. 1996. J. R. Smith and S.-F. Chang, "Visually Searching the Web for Content," IEEE Multimedia � Magazine, Summer, Vol. 4 No. 3, pp.12-20, 1997. M. Flickher, H. Sawhney, W. Niblack, J. Ashley, Q. Huang, B. Dom, M. Gorkani, J. Hafner, � D. Lee, D. Petkovicand D. Steele, and P. Yanker. Query by image and video content: The QBIC system. In IEEE Computer, volume 38, pages 23-31, 1995. Christos Faloutsos, Ron Barber, Myron Flickner, Wayne Niblack, Dragutin Petkovic, and � William Equitz. Efficient and effective querying by image content. J. of Intelligent Information Systems, 3(3/4):231-262, July 1994. (QBIC System) Sikora, T., "The MPEG-7 visual standard for content description-an overview," IEEE � Transactions on Circuits and Systems for Video Technology, Volume: 11 Issue: 6 , Page(s): 696 -702, June 2001. Manjunath, B.S.; Ohm, J.-R.; Vasudevan, V.V.; Yamada, A., "Color and texture � descriptors," IEEE Transactions on Circuits and Systems for Video Technology, Volume: 11 Issue: 6 , Page(s): 703 -715, June 2001. Yossi Rubner, Carlo Tomasi, and Leonidas J. Guibas. A Metric for Distributions with � Applications to Image Databases. Proceedings of the ICCV'98, Bombay, India, January 1998, pages 59-66. Thanks to John R. Smith for some slides on color/texture feature extraction � EE6882-Chang 2
Content-based Image Retrieval System User Query User Query Index Index interface server interface server Image Image User Network User Network thumbnails thumbnails Image/video Image/video Archive Images & Archive Images & Server Server videos videos What functionalities should each component have? What are the bottlenecks of the system? EE6882-Chang 3
Feature Extraction for Content-Based Image Retrieval (Color & Texture) � Why visual features? � Manual annotation is tedious and insufficient � Computers cannot understand images � Comparison of visual features enables comparison of visual scenes � Need tools for organizing filtering and searching through large amounts of visual data � What visual features? � What is available in the data? � What features does the human visual system (HVS) use? � Color: suitable for color images � Texture: visual patterns, surface properties, cues for depth � Shape: boundaries of real world objects, edges � Motion: camera motion vs. object motion EE6882-Chang 4
Visual Features � How to use visual features? � Extraction � Representation � Discrimination � Indexing � Considerations � Complexity � Invariance � Rotation, scaling, cropping, occlusion, shift, etc. � Dimension � Subjective relevance � Distance Metric EE6882-Chang 5
Visual Features (cont.) � Fundamental approach is from pattern recognition work � Group pixels, process the group and generate a feature vector � Discrimination via (transform and ) feature vector distance � Multidimensional indexing of the feature vectors � Do this for color and texture � Build a content-based image retrieval system EE6882-Chang 6
Color Order Systems � The Munsell System (1905) � Colors are arranged so that, as nearly as possible the perceptual distance between adjacent color is constant. The Munsell Book of Color – color chips � The Natural Color System (NCS) – (1981) � Natural Color System Atlas – derived from 60,000 observations � Color are described by the relative amounts of basic colors: black, white, yellow, blue, red and green � The DIN system (1981) � The Coloroid system (1980-1987) � Optical Society of American System (OSA) (1981) � Hunter LAB System (1981) EE6882-Chang 7
Color Order Systems (cont.) � Advantages of Color Order Systems � Easy to understand, plus samples are available � Easy to use and compare colors side-by-side � Number and spacing of samples can be adapted to application � Disadvantages � Too many color order systems, can’t translate between them � Color comparison is only valid for required illuminant � User perception differs � Application to self-luminous colors (i.e., monitors and computer displays) is not easy EE6882-Chang 8
Color Representation � What is COLOR? A weighted combination of stimuli at three principal wavelengths in � the visible spectrum (form blue=400nm to red=700nm). γ ρ β [Oberle] Examples: λ =500nm � ( β , γ , ρ )=(20, 40, 20) B=100 � ( β , γ , ρ )=(100, 5, 4) G=100 � ( β , γ , ρ )=(0, 100, 75) R=100 � ( β , γ , ρ )=(0, 0, 100) EE6882-Chang 9
Tri-stimulus Representation α 1 P 1 ( λ ) α 2 Same Response P 2 ( λ ) HVS ( β , γ , ρ ) α 3 P 3 ( λ ) E.g., use are R, G, B as primary colors P1 , P2 ,P3 Compute correct α 1 α 2 α 3 s.t. the response ( β , γ , ρ ) are the same as those of original color. EE6882-Chang 10
Color Spaces and Color Order Systems � Color Spaces � RGB – cube in Euclidean space R G B = = = r g b + + + + + + R G B R G B R G B � Standard representation used in color displays � Drawbacks � RGB basis not related to human color judgments � Intensity should for one of the dimensions of color � Important perceptual components of color are hue, brightness and saturation EE6882-Chang 11
Color Spaces and Color Order Systems � HSI-cone (cylindrical coordinates) V − = 1 H tan ( 2 ) I 1 / 3 1 / 3 1 / 3 R V = − − V 1 / 6 1 / 6 2 / 6 G 1 1 2 2 = + 1 / 2 S ( V V ) − V 1 / 6 1 / 6 0 B 1 2 2 − − R G 1 2 1 R � Opponent-Cartesian − = − − Bl Y 1 1 2 G − W Bk 1 1 1 B � YIQ-NTSC television standard − − I 0.6 0.28 0.32 R = − Q 0.21 0.52 0.31 G Y 0.3 0.59 0.11 B EE6882-Chang 12
Perceptual Representation Of HSI Space � brightness varies along the vertical axis � hue varies along the circumference � saturation varies along the radius EE6882-Chang 13
Color Coordinate Systems From Jain’s DIP book EE6882-Chang 14
Color Coordinate Systems (cont.) EE6882-Chang 15
Color Space Quantization � How many colors to keep � IBM QBIC � 16M(RGB) � 4096 (RGB) � 64 (Munsell) colors � Columbia U. VisualSEEK � 16M (RGB) � 166 (HSV) colors � (18 Hue, 3 Sat, 3 Val, 4 Gray) � Stricker and Orengo (Similarity of Color Images) � 16M (RGB) � 16 hues, 4 val, 4 sat = 128(HSV) colors � 16M (RGB) � 8 hues, 2 val, 2 sat = 32 (HSV) colors � Sqain and Ballard (Color Indexing) � 16M (RGB) � 8 wb, 16rg, 16by = 2048 (OPP) colors � Independent quantization – each color dimension is quantized independently � Joint quantization – color dimensions are quantized jointly EE6882-Chang 16
Color Histogram � Feature extraction from color images � Choose GOOD color space � Quantize color space to reduce number of colors � Represent image color content using color histogram � Feature vector IS the color histogram = = = 1 if I [ , ] m n r I , [ , ] m n g I , [ , ] m n b ∑∑ R G B = h [ , , ] r g b RGB 0 otherwise m n A color histogram represents the distribution of colors where each histogram bin corresponds to a color is the quantized color space EE6882-Chang 17
Color Histogram (cont.) � Advantages of color histograms � Compact representation of color information � Global color distribution � Histogram distance metrics � Disadvantages � High dimensionality � No information about spatial positions of colors EE6882-Chang 18
Other Histogram Metrics ∑ � L 1 distance + = − D i i ( , 1) H ( ) j H ( ) j + 1 i i 1 j ∑ 2 + = − D i i ( , 1) H ( ) j H ( ) j � L 2 distance + 2 i i 1 ∑ j ( ) min H ( ), j H ( ) j + i i 1 j = − � Histogram Intersection D 1 I ∑ ∑ min H ( ), j H ( ) j + i 1 i j j � Quadratic Distance ∑∑ ( ) ( ) = − α − D H ( j ) H ( j ) ( j , j ) H ( j ) H ( j ) + + Q i 1 i 1 1 1 2 i 2 i 1 2 j j 1 2 α ( j , j ) : correlation between colors j , j e.g. 1-d . � Other histograms 1 2 1 2 j ,j 1 2 � Edge histogram + total edge count � Texture � Issue: quality of edge, texture extraction, lighting (dark frame) EE6882-Chang 19
Recommend
More recommend
