Geometric Registration for Deformable Shapes 3.3 Advanced Global - - PowerPoint PPT Presentation

Geometric Registration for Deformable Shapes

3.3 Advanced Global Matching

Correlated Correspondences [ASP*04] A Complete Registration System [HAW*08]

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

In this session

Advanced Global Matching

• Some practical applications of the optimization presented

in the last session

• Correlated Correspondences [ASP*04]: Applies MRF

model

• A Complete Registration System [HAW*08]: Applies

Spectral matching to filter correspondences

2

Eurographics 2010 Course – Geometric Registration for Deformable Shapes 3

Correlated correspondences

• Correspondence between data and model meshes
• Model mesh is a template; i.e. data is a subset of model
• Not a registration method; just computes corresponding points

between data/model meshes

• Non-rigid ICP [Hanhel et al. 2003] (using the outputted

correspondences) used to actually generate the registration results seen in the paper

Template (Model) Data Result

=

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Basic approach

A joint probability model represents preferred correspondences

• Define a “probability” of each correspondence set

between data/model meshes

• Find the correspondence with the highest probability

using Loopy Belief Propagation (LBP) [Yedidia et al. 2003]

2 main components (next parts of the talk)

• Probability model
• Optimization

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Joint Probability Model

Compatibility constraints

• Involves pair of correspondences
• Represents prior knowledge of which correspondence sets

makes sense

A.

Minimize the amount of deformation induced by the correspondences

B.

Preserve the geodesic distances in model and data

Singleton constraints

• Involves a single correspondence

C.

Corresponding points have same feature descriptor values

5

∏ ∏

=

k k k l k l k kl

c c c Z c P ) ( ) , ( 1 }) ({

,

ψ ψ

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Penalize unnatural deformations

• Edges lengths should stay the same

Compatibility 1: Deformation potential

6 i

x

j

x

In model mesh

ij

l

ij ij

l l ′ ≈

k

z

l

z

Corresponding points in data mesh

ij

l′

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Penalize unnatural deformations

• Edges should twist little as possible
• Is the direction from to in ’s coord system

Compatibility 1: Deformation potential

7 i

x

j

x

j i

d →

j i

d →

i

x

i

x

j

x

i j

d →

i j i j j i j i

d d d d

→ → → →

′ ≈ ′ ≈ ,

In model mesh

j i

d → ′

i j

d → ′

Corresponding points in data mesh

k

z

l

z

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Encoding the preference

• Zero-mean Gaussian noise model for length and twists
• Define potential for each edge in the data mesh
• are “correspondence variables” indicating what is the

corresponding point in the model mesh for respectively

• Caveat: additional rotation needed to measure twist
• For each possibility of precompute aligning rotation

matrices via rigid ICP on surrounding local patch

• Expand corresp. variables to be site/rotation pairs

) | ( ) | ( ) | ( ) , (

i j i j j i j i ij ij l k d

d d G d d G l l G j c i c

→ → → →

′ ′ ′ = = = ψ

d

ψ ) , (

l k z

z ) , (

l k c

c

l k z

z ,

i ck =

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Penalize large changes in geodesic distance

• Geodesically nearby points should stay nearby
• Enforced for each edge in the data mesh

Compatibility 2: Geodesic distance potential

9 i

x

j

x

Corresponding points in model mesh

) , (

j i Geodesic

x x dist

k

z

l

z

Adjacent points in data mesh If > 3.5p  prob assigned 0

• therwise  prob assigned 1

p z z dist

l k Geodesic

≈ ) , (

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Penalize large changes in geodesic distance

• Geodesically far points should stay far away
• Enforced for each pair of points in the data mesh whose

geodesic distance is > 5p

10 i

x

j

x

Corresponding points in model mesh

) , (

j i Geodesic

x x dist

k

z

l

z

Adjacent points in data mesh If < 2p  prob assigned 0

• therwise  prob assigned 1

p z z dist

l k Geodesic

5 ) , ( >

Compatibility 2: Geodesic distance potential

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Singletons: Local surface signature potential

Spin images gives matching score for each individual correspondence

• Compute spin images & compress using PCA

 gives surface signature at each point

• Discrepancy between (data) and (model)
• Zero-mean Gaussian noise model

11 i

x

i

x

s

k

z

s

i

x

s

i

x

k

z

Compare Spin Images

k

z

s

i

x

s

Model mesh Data mesh

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Model summary

Get Pairwise Markov Random Field (MRF)

• Pointwise potential for each pt in data
• Pairwise potential for each edge in data
• Far geodesic potentials for each pair of points > 5p apart

Model mesh Data mesh

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Quick intro: Markov Random Fields

Joint probability function visualized by a graph

• Prob. = Product of the potentials at all edges

13

“Observed” nodes “Hidden” nodes

) , (

l k kl

c c ψ ) ( k

k c

ψ

 (ex) Surface signature potential  (ex) Deformation, geodesic distance potential

∏ ∏

=

k k k l k l k kl

c c c Z c P ) ( ) , ( 1 }) ({

,

ψ ψ

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Loopy Belief Propagation (LBP)

Compute marginal probability for each variable

• Pick variable value that maximizes the marginal prob.

Usual way to compute marginal probabilities (tabulate and sum up) takes exponential time

• BP is a dynamic programming approach to efficiently

compute marginal probabilities

• Exact for tree MRFs, approximate for general MRFs

14

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Basic idea

• Marginals at node proportional to product of pointwise

potential and incoming messages

Loopy Belief Propagation (LBP)

15 i

x

c

c

b

c

a

c

d

c

k a

m →

k b

m →

k c

m →

k d

m →

∏

∈ →

=

) (

) ( ) ( ) (

k N l k k l k k k k

c m c k c b φ

k

c

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Basic idea

• Compute these messages (at each edge) and we are done

Loopy Belief Propagation (LBP)

16

∑

l

c

• f

values all l k kl l l

c c c ) , ( ) ( ψ φ

l

c

j

x

∏ ∑

∈ → →

←

k l N q l l q c

• f

values all l k kl l l k k l

c m c c c c m

l

\ ) (

) ( ) , ( ) ( ) ( ψ φ

k l

m →

k

c

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Basic idea

• Compute these messages (at each edge) and we are done

Loopy Belief Propagation (LBP)

17 c

c

b

c

a

c

d

c

l a

m →

l b

m →

l c

m →

l d

m →

∑

l

c

• f

values all l k kl l l

c c c ) , ( ) ( ψ φ

l

c

j

x

∏ ∑

∈ → →

←

k l N q l l q c

• f

values all l k kl l l k k l

c m c c c c m

l

\ ) (

) ( ) , ( ) ( ) ( ψ φ

k l

m →

k

c

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Basic idea

• Compute these messages (at each edge) and we are done
• Recursive formulation
• Start at ends and work your way towards the rest

Loopy Belief Propagation (LBP)

18 c

c

b

c

a

c

d

c

l a

m →

l b

m →

l c

m →

l d

m →

∑

l

c

• f

values all l k kl l l

c c c ) , ( ) ( ψ φ

l

c

j

x

∏ ∑

∈ → →

←

k l N q l l q c

• f

values all l k kl l l k k l

c m c c c c m

l

\ ) (

) ( ) , ( ) ( ) ( ψ φ

k l

m →

k

c

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Loops: iterate until messages converge

• Start with initial values (ex: )
• Apply message update rule until convergence
• Convergence not guaranteed, but works well in practice

Loopy Belief Propagation (LBP)

19

∑

a

c

• f

values all b a ab a a

c c c ) , ( ) ( ψ φ

c

c

b

c

a

c

b a

m →

a b

m →

a c

m →

c a

m →

b c

m →

c b

m →

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Results & Applications

• Efficient, coarse-to-fine implementation
• Xeon 2.4 GHz CPU, 1.5 mins for arm, 10 mins for puppet

Correspondences on human body models Finding articulated parts Interpolation between poses

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Next topic: HAW*08

An application to the spectral matching method of last session

• A good illustration of how a matching method fits into a

real registration pipeline

A pairwise method

• Deform the source shape to match the target shape

Gray = source Yellow = target

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Overview

Performs both correspondence and deformation

• Correspondences based on improving closest points
• After finding correspondences, deform to move shapes

closer together

• Re-take correspondences from the deformed position
• Deform again, and repeat until convergence

22

Correspondence Deformation

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Overview

Performs both correspondence and deformation

23

5 basic steps

1.Closest points 2.Improve by feature matching 3.Filter by spectral matching 4.Expand sparse set 5.Fine-tune target locations

Correspondence Deformation

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Overview

Performs both correspondence and deformation

24

2 basic steps 1.Fit per-cluster rigid transformation 2.Sparse least-squares solve for deformed positions Occasional step: Increase cluster size Correspondence Deformation

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Detailed Overview

Sampling

• Whole process works with reduced sample set

Correspondence & Deformation

• Examine each step in more detail

Discussion

• Discuss pros/cons

25

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Sample for robustness & efficiency

Coarse to fine approach

• Use uniform subsampling of the surface and its normals
• Improve efficiency, can improve robustness to local

minima

Let’s make it more concrete

• Sample set denoted
• In correspondence: for each , find corresponding target

points

• In deformation: given , find deformed sample positions

that match while preserving local shape detail

26

i

s

i

s′

i

t

i

s

i

t

i

t

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Correspondence Step #1

Find closest points

• For each source sample, find

the closest target sample

• s = sample point on source
• t = sample point on target
• Usually pretty bad

Target (yellow)  Source (gray)

27

2 ˆ

min arg t s

T t

−

∈

Closest point correspondences

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Correspondence Step #2

Improve by feature matching

• Search target’s neighbors to

see if there’s better feature match, replace target

• Let f(s) be feature value of s
• Iterate until we stop moving
• If we move too much, discard

correspondence

• Much better, but still outliers

Target (yellow)  Source (gray)

28

2 ) (

) ( ) ( min arg t f s f t

t N t

′ − ←

∈ ′

Feature-matched correspondences

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Correspondence Step #3

Filter by spectral matching

• (First some preprocessing)
• Construct k-nn graph on both

src & tgt sample set (k = 15)

• Length of shortest path on

graph gives approx. geodesic distances on src & tgt

• Goal is to filter these ----------

and keep a subset which is geodesically consistent

29

Target (yellow)  Source (gray) Feature-matched correspondences

) , ( ) , (

j i g j i g

t t d s s d

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Correspondence Step #3

Filter by spectral matching

• Construct affinity matrix M

using these shortest path distances

• Consistency term & matrix
• Threshold c0 = 0.7 gives how

much error in consistency we are willing to accept

30

1 }, ) , ( ) , ( , ) , ( ) , ( min{ = =

ii j i g j i g j i g j i g ij

c s s d t t d t t d s s d c

Target (yellow)  Source (gray) Feature-matched correspondences

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Correspondence Step #3

Filter by spectral matching

• Apply spectral matching: find

eigenvector with largest eigenvalue  score for each correspondence

• Iteratively add corresp. with

largest score while consistency with the rest is above c_0

• Gives kernel correspondences
• Filtered matches usually sparse

Target (yellow)  Source (gray)

31

Filtered correspondences

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Correspondence Step #4

Expand sparse set

• Lots of samples have no target

position

• For these, find best target

position that respects geodesic distances to kernel set

Target (yellow)  Source (gray)

32

Expanded correspondences

2 ( , )

( , ) ( , ) ( , )

k k

K g k g k K

e d d

∈

  = −  

∑

s t

s t s s t t

( , )

arg min ( , )

g j

i K i N T

e

∈

=

t t

t s t

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Correspondence Step #4

Expand sparse set

• Lots of samples have no target

position

• Compute confidence weight

based only how well it respects geodesic distances to kernel set

Target (yellow)  Source (gray)

33

Expanded correspondences

Red = not consistent --- Blue = very consistent ( , ) exp( ) 2

K i i i

e w e = − s t

( , )

1 ( , )

k k

K k k K

e e K

∈

=

∑

s t

s t

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Correspondence Step #5

Fine-tuning

• So far, target points restricted

to be points in target samples

• Not accurate when shapes are

close together

• Relax this restriction and let

target points become any point in the original point cloud

• Replace target sample with a

closer neighbor in the original point cloud

Target (yellow)  Source (gray)

34

Expanded correspondences

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Deformation

Solved by energy minimization (least squares)

• Last step gave target positions
• Now find deformed sample positions that match

target positions

Two basic criteria:

• Match correspondences: should be close to
• Shape should preserve detail (as-rigid-as-possible)
• Combine to give energy term:

35

corr corr rigid rigid

E E E λ λ = +

i

s′

i

s

i

t

i

t

i

t

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Correspondence matching term

Combination of point-to-point (α=0.6) and point-to- plane (β=0.4) metrics

• Weighted by confidence weight wi of the target position

36

2 ' ' 2

(( ) )

i

T corr i i i i i i S

E w α β

∈

  = − + −    

∑

s

s t s t n

Point-to-point Point-to-plane

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Shape preservation term

Deformed positions should preserve shape detail

• Form an extended cluster for each sample point: the

sample itself and its neighbors

• For each find the rigid transformation (R,T) from

sample positions to their deformed locations

• When solving for , constrain them to move rigidly

according to each cluster that it’s associated with

37

k

C ~

2 '

i k

k k i k i s C

E

∈

= + −

∑ R s

T s

k

C ~

i

s′

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Clusters for local rigidity

• Initially each cluster contains a single sample point
• Every 10 iterations (of correspondence & deformation),

combine clusters that have similar rigid transformations (forming larger rigid parts)

38

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Advantages of features & clustering

39

Source + Target Without Features Without Clustering With Both

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Results

40

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Results

41

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Results

Efficient, robust method

42

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Conclusion

Correlated correspondence

• Robust method for matching correspondences
• Measure how much the correspondence “makes sense”
• Probability model  optimized using LBP
• Requires a template
• If model is incomplete, then there is no “correct” corresponding

point to assign

Eurographics 2010 Course – Geometric Registration for Deformable Shapes

Conclusion

Non-rigid registration under isometric deformations

• Improve closest point correspondences using features and

spectral matching

• Deform shape while preserving local rigidity of clusters
• Iteratively estimate correspondences and deformation

until convergence

• Robust, efficient method
• Relies on geodesic distances (problematic when holes are

too large)

44