Announcements Write your CS login ID on the pset hardcopy Texture - PDF document

9/15/2009 Announcements • Write your CS login ID on the pset hardcopy Texture Tuesday, Sept 15 Kristen Grauman UT-Austin Review: last time Texture • Edge detection: – Filter for gradient – Threshold gradient magnitude, thin • Binary image analysis Bi i l i – Connected components to find regions – Morphological operators to “clean up” What defines a texture? Includes: more regular patterns Includes: more random patterns 1

9/15/2009 Shape from texture Texture-related tasks • Use deformation of texture from point to point to • Shape from texture estimate surface shape – Estimate surface orientation or shape from image texture Pics from A. Loh: http://www.csse.uwa.edu.au/~angie/phdpics1.html Analysis vs. Synthesis Texture-related tasks • Shape from texture Why analyze – Estimate surface orientation or shape from texture? image texture • Segmentation/classification from texture cues – Analyze, represent texture – Group image regions with consistent texture • Synthesis – Generate new texture patches/images given some examples Images:Bill Freeman, A. Efros 2

9/15/2009 What kind of response will we get with an edge get with an edge detector for these images? Images from Malik and Perona, 1990 http://animals.nationalgeographic.com/ Why analyze texture? Importance to perception: • Often indicative of a material’s properties • Can be important appearance cue, especially if shape is similar across objects • Aim to distinguish between shape, boundaries, and texture Technically: • Representation-wise, we want a feature one step above “building blocks” of filters, edges. …and for this image? Image credit: D. Forsyth Psychophysics of texture • Some textures distinguishable with preattentive perception– without scrutiny, eye movements [Julesz 1975] Same or different? 3

9/15/2009 Capturing the local patterns with Texture representation image measurements • Textures are made up of repeated local [Bergen & patterns, so: Adelson, – Find the patterns Nature 1988] • Use filters that look like patterns (spots, bars, raw patches…) patches…) S Scale of l f • Consider magnitude of response patterns – Describe their statistics within each local influences window discriminability • Mean, standard deviation • Histogram Size-tuned linear filters • Histogram of “prototypical” feature occurrences Texture representation: example Texture representation: example mean mean mean mean d/dx d/dy d/dx d/dy value value value value Win. #1 4 10 Win. #1 4 10 Win.#2 18 7 original image original image … … statistics to statistics to summarize patterns summarize patterns derivative filter derivative filter in small windows in small windows responses, squared responses, squared 4

9/15/2009 Texture representation: example Texture representation: example mean mean mean mean d/dx d/dy d/dx d/dy value value value value Win. #1 4 10 Win. #1 4 10 Win.#2 18 7 Win.#2 18 7 … Win.#9 20 20 original image original image … … statistics to statistics to summarize patterns summarize patterns derivative filter derivative filter in small windows in small windows responses, squared responses, squared Texture representation: example Texture representation: example Windows with primarily horizontal Both edges mean d/dy value) mean d/dy value) mean mean mean mean d/dx d/dy d/dx d/dy value value value value Win. #1 4 10 Win. #1 4 10 Dimension 2 (m Dimension 2 (m Win.#2 18 7 Win.#2 18 7 … … Win.#9 20 20 Win.#9 20 20 Dimension 1 (mean d/dx value) … Dimension 1 (mean d/dx value) … Windows with Windows with statistics to small gradient in statistics to primarily vertical summarize patterns summarize patterns both directions edges in small windows in small windows Texture representation: example Texture representation: example mean d/dy value) mean mean d/dx d/dy value value Far: dissimilar Win. #1 4 10 textures Dimension 2 (m Win.#2 18 7 Close: similar … textures Win.#9 20 20 original image … Dimension 1 (mean d/dx value) visualization of the assignment to texture “types” statistics to summarize patterns derivative filter in small windows responses, squared 5

9/15/2009 Texture representation: example Texture representation: example = − + − 2 2 D ( a , b ) ( a b ) ( a b ) 1 1 2 2 a a a 2 ∑ nsion 2 nsion 2 = − 2 ( , ) ( ) D a b a b i i = b b b b i i 1 1 b b Dime Dime Dimension 1 Dimension 1 b Distance reveals how dissimilar texture from window a is from texture in window b. Texture representation: Filter banks window scale • Our previous example used two filters, and • We’re assuming we know the relevant window resulted in a 2-dimensional feature vector to size for which we collect these statistics. describe texture in a window. – x and y derivatives revealed something about local Possible to perform scale Possible to perform scale structure. t t selection by looking for • We can generalize to apply a collection of window scale where texture multiple ( d ) filters: a “filter bank” description not changing. • Then our feature vectors will be d -dimensional. – still can think of nearness, farness in feature space Multivariate Gaussian Filter banks orientations “Edges” scales “Bars” “Spots” Spots • What filters to put in the bank? – Typically we want a combination of scales and orientations, different types of patterns. ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ 9 0 10 5 16 0 Σ = Σ = Σ = ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ 5 5 ⎦ ⎣ 0 9 ⎦ ⎣ ⎦ 0 9 Matlab code available for these examples: http://www.robots.ox.ac.uk/~vgg/research/texclass/filters.html 6

9/15/2009 Filter bank tincap2.jpg Image from http://www.texasexplorer.com/aus Showing magnitude of responses 7

9/15/2009 8

9/15/2009 9

9/15/2009 [r1, r2, …, r38] We can form a feature vector from the list of responses at each pixel. d -dimensional features General definition of d ∑ = − 2 D ( a , b ) ( a b ) inter-point Euclidean i i Example uses of = distance (L 2 ). i 1 texture in vision: texture in vision: analysis . . . 3d 2d 10

9/15/2009 Classifying materials, “stuff” Texture features for image retrieval Y. Rubner, C. Tomasi, and L. J. Guibas. The earth mover's distance as a metric for image retrieval. International Journal of Computer Vision , Figure by Varma 40(2):99-121, November 2000, & Zisserman Characterizing scene categories by texture Segmenting aerial imagery by textures L. W. Renninger and J. Malik. When is scene identification just texture recognition? Vision Research 44 (2004) 2301–2311 http://www.airventure.org/2004/gallery/images/073104_satellite.jpg Texture synthesis Texture-related tasks • Goal: create new samples of a given texture • Shape from texture • Many applications: virtual environments, hole- – Estimate surface orientation or shape from filling, texturing surfaces image texture • Segmentation/classification from texture cues – Analyze, represent texture – Group image regions with consistent texture • Synthesis – Generate new texture patches/images given some examples 11

9/15/2009 Markov Chains The Challenge Markov Chain • a sequence of random variables • is the state of the model at time t repeated • Need to model the whole spectrum: from repeated to spectrum: from repeated to • Markov assumption : each state is dependent only on the stochastic texture previous one – dependency given by a conditional probability : stochastic Alexei A. Efros and Thomas K. Leung, “Texture Synthesis by Non-parametric Sampling,” Proc. • The above is actually a first-order Markov chain International Conference on Computer Vision (ICCV), 1999. • An N’th-order Markov chain: Both? Source S. Seitz Markov Chain Example: Text Text synthesis “A dog is a man’s best friend. It’s a dog eat dog world out there.” Create plausible looking poetry, love letters, term papers, etc. Most basic algorithm a 2/3 1/3 1. Build probability histogram dog 1/3 1/3 1/3 – find all blocks of N consecutive words/letters in training documents is 1 – compute probability of occurrence man’s 1 2. Given words best 1 – compute by sampling from compute by sampling from friend friend 1 1 it’s 1 eat 1 WE NEED TO EAT CAKE world 1 out 1 there 1 . 1 a dog . is man’s best friend it’s eat out there world Source: S. Seitz Source: S. Seitz Text synthesis Text synthesis Synthesizing Computer Vision text • Results: – “As I've commented before, really relating to someone involves standing next to • What do we get if we impossible.” extract the probabilities – "One morning I shot an elephant in my g p y from the F&P chapter on p arms and kissed him.” Linear Filters, and then synthesize new – "I spent an interesting evening recently statements? with a grain of salt" Dewdney, “A potpourri of programmed prose and prosody” Scientific American, 1989. Check out Yisong Yue’s website implementing text generation: build your own text Markov Chain for a given text corpus. http://www.yisongyue.com/shaney/index.php Slide from Alyosha Efros, ICCV 1999 12