Mathematical and Perceptual Models for Image Segmentation Thrasos - PowerPoint PPT Presentation

Mathematical and Perceptual Models for Image Segmentation Thrasos Pappas Electrical & Computer Engineering Department Northwestern University pappas@ece.northwestern.edu http://www.ece.northwestern.edu/~pappas Banff, July 27, 2005

People ! Junqing Chen, Unilever Research ! Dejan Depalov, Northwestern University ! Aleksandra Mojsilovic, IBM T.J. Watson Research Center ! Bernice Rogowitz, IBM T.J. Watson Research Center ! Dongge Li, Motorola Labs ! Bhavan Gandhi, Motorola Labs Thrasos Pappas, Banff, July 27, 2005 2

Problem Images “Ideal” Segmentations Semantic Categories landscape sky mountain water forest sky forest cityscape manmade people outdoor Thrasos Pappas, Banff, July 27, 2005 3

Semantic Information Extraction ! Motivation – Proliferation of image and video acquisition devices (digital still and video cameras, image and video phones, PDAs) – World rich in digital visual content – Large personal repositories (consumer market) – Increasing processing capabilities ! Goal: Intelligent content management – Semantic labeling – Content organization – Efficient retrieval ! Techniques – Image and video segmentation – Extracting semantically related features – Relating features to semantic categories Thrasos Pappas, Banff, July 27, 2005 4

Challenges ! What are the important semantic categories? ! How to link the low-level features to semantically important categories? Thrasos Pappas, Banff, July 27, 2005 5

Semantic Categories ! Recent perceptual experiments by Mojsilovic and Rogowitz identified important semantic categories that humans use for image classification Man-made Less human-like Natural More human-like ! Conjecture: Semantic categories can be derived from combinations of low-level image features Thrasos Pappas, Banff, July 27, 2005 6

Bridging the Semantic Gap Semantics High level Use segment descriptors and statistical techniques to relate segments (first) and scenes (later) to semantic categories/labels Perceptually Uniform Medium level Segments Incorporate knowledge of human perception and image characteristics into feature extraction and algorithm design Low level Primitives Thrasos Pappas, Banff, July 27, 2005 7

Adaptive Clustering Algorithm

Adaptive Clustering Algorithm K-means Class Labels ACA Class Labels Original Image Thrasos Pappas, Banff, July 27, 2005 9

Adaptive Clustering Algorithm (ACA) ! K-means clustering (LBG) – Based on image histogram – No spatial constraints – Each cluster is characterized by constant intensity ! Add spatial constraints – Region model: Markov/Gibbs random field ! Make it adaptive – Cluster centers spatially varying – Texture model: spatially varying mean + WGN ! MAP estimates of segmentation x given observation y p ( x | y ) p ( y | x ) p ( x ) ∝ Thrasos Pappas, Banff, July 27, 2005 10

ACA ! K-means minimizes ' x 2 ( y ) − µ s s s ! Adaptive clustering maximizes & # 1 ' ' x 2 p ( x | y ) exp ( y ) V ( x ) % " ∝ − − µ s − s s C 2 2 $ ! σ s C ! Or, minimizes 1 ' ' x 2 ( y ) V ( x ) − µ s + s s C 2 2 σ s C Thrasos Pappas, Banff, July 27, 2005 11

ACA: Local Intensity Function Estimation x ! Given , segmentation into classes ! Estimate x , ∀ x s , s s µ s Intensity function for each class at each point in the image ! Use hierarchy of window sizes Thrasos Pappas, Banff, July 27, 2005 12

ACA Thrasos Pappas, Banff, July 27, 2005 13

ACA: Region Estimation x , ∀ x s , s s ! Given µ s p ( x | y ) ! Maximize (too difficult) ! Maximize marginal densities (Iterated Conditional Modes) p ( x | y , x , q s ) p ( x | y , x , q N ) ∀ ≠ = ∈ s q s s q s Thrasos Pappas, Banff, July 27, 2005 14

K-means vs. ACA Thrasos Pappas, Banff, July 27, 2005 15

K-means Clustering Thrasos Pappas, Banff, July 27, 2005 16

K-means Clustering Thrasos Pappas, Banff, July 27, 2005 17

ACA: Local Intensity Functions (15x15) Thrasos Pappas, Banff, July 27, 2005 18

ACA: Model (15x15) Thrasos Pappas, Banff, July 27, 2005 19

Adaptive Clustering Algorithm Original Image ACA Class Labels ACA Model (7x7) Thrasos Pappas, Banff, July 27, 2005 20

Adaptive Clustering Algorithm Original Image ACA Class Labels ACA Model (15x15) Thrasos Pappas, Banff, July 27, 2005 21

Adaptive Clustering Algorithm Original Image ACA Class Labels ACA Model (31x31) Thrasos Pappas, Banff, July 27, 2005 22

Image Restoration Models ! Simple space varying image model [Kuan et al.` 85] – Space-varying mean + white Gaussian noise ! Spatially-adaptive LMMSE estimator – Use local sample mean and local sample variance ! No explicit model for region boundaries – Computes sample mean/variance across boundaries Thrasos Pappas, Banff, July 27, 2005 23

K-means vs. ACA Thrasos Pappas, Banff, July 27, 2005 24

ACA Thrasos Pappas, Banff, July 27, 2005 25

Adaptive Perceptual Color-Texture Segmentation

Natural Textures ! Combine color composition, spatial characteristics ! Non-uniform statistical characteristics (lighting, perspective) ! Perceptually uniform ! Need spatially adaptive features ! Small number of parameters Thrasos Pappas, Banff, July 27, 2005 27

Texture Synthesis [Portilla-Simoncelli’00] Thrasos Pappas, Banff, July 27, 2005 28

Adaptive Perceptual Color-Texture Segmentation ← Slowly varying Dominant Colors Color Composition Feature Extraction Original Spatial Texture Final segmentation Feature Extraction ← Texture Class Labels Grayscale Thrasos Pappas, Banff, July 27, 2005 29

Dominant Colors ! Human eye cannot simultaneously perceive a large number of colors – Even though, under appropriate adaptation, it can distinguish more than 2M colors ! Small set of color categories – Efficient representation – Easier to capture invariant properties of object appearance ! Color categories are related statistical structure of perceived environment – K-means clustering to compute color categories [Yendrikovskij’00] Thrasos Pappas, Banff, July 27, 2005 30

Spatially Adaptive Dominant Colors ! Dominant colors [Ma’97, Mojsilovic’00] – For class of images – For a given image ! Current approaches to extract dominant colors: – K-means (VQ) [LBG’80]; – Mean-shift [Comaniciu-Meer’97]; Assumption: constant dominant colors ! Proposed approach: – Spatially adaptive dominant colors – Use ACA Thrasos Pappas, Banff, July 27, 2005 31

Comparison with Mean-Shift 4 colors ACA Original Image quantization over-segmentation under-segmentation Thrasos Pappas, Banff, July 27, 2005 32

Color Composition Feature ! Constant Dominant Colors: c : color [ ] { ( ) } f c , p , i 0 , , n , p 0 , 1 i ! = = ∈ p c i i i : percentage i ! Spatially Adaptive Dominant Colors: [ ] { ( ) } f ( s , N ) c , p , i 0 , , n , p 0 , 1 ! = = ∈ c s i i i ! ACA adapts to local characteristics. ! Dominant colors relatively constant in small neighborhood: Can approximate with intensity at center of window. Thrasos Pappas, Banff, July 27, 2005 33

Color Feature Similarity Metric ! Optimal Color Composition Distance (OCCD) [Mojsilovic’00] – Quantize color component based on percentage – Find best color correspondence – Then compute distance as sum of distances between matched colors (in a given colorspace) Thrasos Pappas, Banff, July 27, 2005 34

Illustration of OCCD computation A :( ,30) ( ,30) ( ,20) ( ,20) • Color Quantization unit p = 10 B :( ,40) ( ,30) ( ,30) • Weight of the link is C max -cost (color distance in Lab color A : space, C max =376) B : • Solve maximum graph matching problem using Gabow’s algorithm. A : 131 • Apply color metric to resulting 0 graph. 61 55 30 B : OCCD dist = 61*.3+55*.2+30*.1+131*.1=45.4 Thrasos Pappas, Banff, July 27, 2005 35

Spatial Texture Features ! Grayscale image component (vs. achromatic pattern map) ! Multiscale frequency decomposition – DWT (9/7 Daubechies) – Steerable filters [Freeman-Adelson’91] – Gabor filters [Daugman’86] ! Energy of subband coefficients is sparse – Use local median energy Thrasos Pappas, Banff, July 27, 2005 36

Steerable Pyramid Decomposition π − π π − π Ideal spectrum Ideal spectrum 1-level decomposition 2-level decomposition Thrasos Pappas, Banff, July 27, 2005 37

Steerable Pyramid Decomposition π − π π − π Ideal spectrum Actual spectrum Thrasos Pappas, Banff, July 27, 2005 38

Smooth vs. Non-smooth Classification ! For each pixel: – S max = Maximum of 4 subband responses – S i = Index of maximum coefficients – Local median energy extraction on S max – 2-level K-means on local median (Check validity of smooth/non-smooth cluster) – Use threshold provided by subjective test Thrasos Pappas, Banff, July 27, 2005 39