Random Graph Models for Image Patches Franois Meyer University of - PowerPoint PPT Presentation

Random Graph Models for Image Patches François Meyer University of Colorado at Boulder Joint work with Kye Taylor February 7, 2014 ICERM, Spring 2014; Research Cluster: Geometric analysis methods for graph algorithms

Outline 1 Introduction 2 Notations, definitions 3 Warm-up: a first look at the patch set 4 The graph models 5 A better distance on the graph 6 Commute time on the graph models 7 From the random walk to the eigenvectors François Meyer | Random Graph Models for Image Patches 2/55

1 Introduction François Meyer | Random Graph Models for Image Patches 3/55

A fresh perspective on image processing 1. break the image into pieces (jigsaw puzzle) 2. piece = patch 3. gather the pieces that look alike and do something with them 4. reconstruct the jigsaw puzzle François Meyer | Random Graph Models for Image Patches 4/55

Patch-Graph idea: state-of-the art image processing Denoising • nonlocal means algorithm [Buades et al., 2005] • learn the local geometry of the set of patches using SVDs [Elad and Aharon, 2006, Dabov et al., 2009, Guleryuz, 2007] • denoise by applying a diffusion on the graph of “patches” [ Szlam et al., 2008 , Hein and Maier, 2007 , Bougleux et al., 2009 ] • analysis of the Fokker-Planck diffusion operator [Singer et al., 2009] Inpainting, Hyper-resolution • fill in regions with similar patches [Criminisi et al., 2004, Zontak and Irani, 2011] François Meyer | Random Graph Models for Image Patches 5/55

Problem statement and contribution Goal: understand the geometry observed in general patch-graphs Line of attack: 1. prototypical graph models that epitomize this geometry • patches within which the intensity varies smoothly, • patches where the intensity exhibits very rapid changes 2. probabilistic analysis of the commute time metric 3. spectral decomposition of commute time → predict the shape of the eigenvectors of the graph Laplacian François Meyer | Random Graph Models for Image Patches 6/55

2 Notations, definitions François Meyer | Random Graph Models for Image Patches 7/55

Image patch √ √ Definition 1 Let u be a N × N image. Let x n = ( i, j ) be a pixel with √ linear index n = i × N + j . We extract an m × m block, centered about x n ,  · · ·  u ( i − m/2, j − m/2 ) u ( i − m/2, j + m/2 ) . . . .  ,   . .  u ( i + m/2, j − m/2 ) · · · u ( i + m/2, j + m/2 ) We identify the m × m matrix with a vector in R m 2 , and we define the patch     u 1 ( x n ) u ( i − m/2, j − m/2 ) . . . . u ( x n ) =  =  .     . .   u m 2 ( x n ) u ( i + m/2, j + m/2 ) François Meyer | Random Graph Models for Image Patches 8/55

The patch-set Definition 2 The patch-set is defined as the set of patches extracted from the image u , P = { u ( x n ) , n = 1, 2, . . . , N } . 1 3 2 6 2 7 7 1 5 4 3 6 5 4 The patch-graph François Meyer | Random Graph Models for Image Patches 9/55

Definition 3 The patch-graph , Γ , is a weighted graph defined as follows. 1. The vertices of Γ are the N patches u ( x n ) , n = 1, . . . , N . 2. Each vertex u ( x n ) is connected to its ν nearest neighbors using the metric d ( n, m ) = � u ( x n ) − u ( x m ) � + β � x n − x m � . 3. The weight w n,m along the edge { u ( x n ) , u ( x m ) } is given by  e − d 2 ( n, m ) /σ 2  if x n is connected to x m , w n,m =  0 otherwise. François Meyer | Random Graph Models for Image Patches 10/55

3 Warm-up: a first look at the patch set François Meyer | Random Graph Models for Image Patches 11/55

Local variance within a patch H I H I François Meyer | Random Graph Models for Image Patches 12/55

Projection of the patches in R 3 H I • smooth patches within which u ( x ) varies smoothly are clustered along low dimensional curves and surfaces • rough patches within which �∇ u ( x ) � is large are shattered all over the patch-set François Meyer | Random Graph Models for Image Patches 13/55

Mutual distances: weight matrix W apply a permutation τ of the patch indices: Var ( x τ ( 1 ) ) � Var ( x τ ( 2, ) � · · · � Var ( x τ ( N ) ) . H I • smooth patches are close to one another • rough patches are at a large distance of one another François Meyer | Random Graph Models for Image Patches 14/55

Distribution of distances between patches: more experimental evidence [Zontak and Irani, 2011] distance to the nearest neighbor of a patch: grows exponentially with the gradient within that patch François Meyer | Random Graph Models for Image Patches 15/55

Distribution of distances between patches: more experimental evidence • smooth patches within which u ( x ) varies smoothly are clustered along low dimensional curves and surfaces • rough patches within which �∇ u ( x ) � is large are shattered all over the patch-set • ... and yet rough patches and smooth patches to not talk to one another “Smooth patches are all alike; every rough patch is rough in its own way” François Meyer | Random Graph Models for Image Patches 16/55

Patch graph: community structure? 0.8 low to high low to low 0.6 high to high 0.4 0.2 0 A B C D E F G H I probability that an edge [ n, m ] connects patches x n and x m • same variance: low or high • different variance → two communities that are weakly connected. François Meyer | Random Graph Models for Image Patches 17/55

We need a better distance principal component analysis of the patch set from the butterfly image − − − − − − − − − − ...but where is the beautiful structure? − ⇒ we seek a better distance on the patch-set − Idea: when physical distances are very different, a notion of connectivity may be more useful François Meyer | Random Graph Models for Image Patches 18/55

Summary and Line of attack • experimental observation about the mutual distance in the dataset → change the metric on the graph • construct a prototypical graph model that epitomizes the experi- mental patch-graphs – a smooth subgraph of smooth patches ∈ smooth regions – a rough subgraph of rough patches ∈ edges, texture ⇒ predict the shape of the eigenfunctions φ k François Meyer | Random Graph Models for Image Patches 19/55

4 The graph models François Meyer | Random Graph Models for Image Patches 20/55

The prototypical graph models • Goal: predict the shape of the eigenfunctions φ k on the patch-set from the knowledge of the commute time • compute estimates of the commute time on simple graph models • the graph models epitomize the characteristic features observed in patch-graphs • the graph model is composed of two subgraphs: – a smooth subgraph of smooth patches ∈ smooth regions – a rough subgraph of rough patches ∈ edges, texture François Meyer | Random Graph Models for Image Patches 21/55

The graph model Mixture of smooth and rough patches: combine a smooth and a rough subgraph of equal size Definition 4 The graph Γ ∗ ( N ) is a weighted graph composed of a smooth subgraph S ( N/2, B ) and a rough subgraph R ( N/2, p ) . Edges between S ( N/2, B ) and R ( N/2, p ) are created randomly and in- dependently with probability q and assigned the edge weight w c > 0 . F (N/2,p) S (N/2,L) Γ ∗ ( N ) W François Meyer | Random Graph Models for Image Patches 22/55

The smooth subgraph model • smooth patches: large entry W n,m when | n − m | is small • spatial proximity ⇒ proximity in patch-space Definition 5 The smooth graph S ( N, B ) is a weighted graph com- posed of N vertices, x 1 , . . . , x N . The weight on the edge { x n , x m } is defined by � if | n − m | � B, w S 1 � n, m � N w n,m = for 0 otherwise, • weight w S > 0 : distance between two spatially adjacent patches • B = thickness of the diagonal in W François Meyer | Random Graph Models for Image Patches 23/55

The rough subgraph model • rough patches: large entries in W scattered throughout the matrix • spatial proximity � proximity in patch space Definition 6 The rough graph R ( N, p ) is a random weighted graph com- posed of N vertices, x 1 , . . . , x N . The weight on the edge { x n , x m } is defined by � w R with probability p, if 1 � n < m � N, w n,m = with probability 1 − p 0 and w n,m = 1 if n = m . • weighted Erdös-Renyi graph with self-connections. • p controls the density of the edges François Meyer | Random Graph Models for Image Patches 24/55

5 A better distance on the graph François Meyer | Random Graph Models for Image Patches 25/55

How fast can one diffuse on the patch-set? • random walk Z k on the vertices of the patch-graph with transition probability matrix P = W n,m w n,m P n,m = Prob ( Z k + 1 = x m | Z k = x n ) � . � l w n,l D n,n • start the random walk at x n and count the number of steps necessary to reach x m : hitting-time h ( x n , x m ) = E n min { j � 0 : Z j = x m } , • commute time: symmetric hitting time κ ( x n , x m ) = h ( x n , x m ) + h ( x m , x n ) . François Meyer | Random Graph Models for Image Patches 26/55