Learning Dense Correspondence via 3D-guided Cycle Consistency - PowerPoint PPT Presentation

Learning Dense Correspondence via 3D-guided Cycle Consistency Tinghui Zhou 1 , Philipp Krähenbühl 1 , Mathieu Aubry 2 , Qixing Huang 3 , Alexei A. Efros 1 UC Berkeley 1 , ENPC ParisTech 2 , TTI-Chicago 3

The Unreasonable Effectiveness of Deep Learning? Performance gain over traditional methods 60% Lots of Very few direct labels direct labels 45% 30% 15% 0 Object Semantic Human Intrinsic Video Dense detection seg. pose image Seg. matching

Dense Semantic Correspondence 3

Dense Semantic Correspondence 4

Traditional Pairwise Methods Hand-crafted Features Feature Matching Hand-crafted Features • SIFT flow : Liu et al. , ECCV 2008 • Generalized PatchMatch : Barnes et al. , ECCV 2010 • Deformable Spatial Pyramid : Kim et al. , CVPR 2013 5

Collection Correspondence Congealing : Learned-Miller, PAMI 2006 • Collection Flow : Kramelmacher-Shlizerman et al. , CVPR 2012 • Object discovery and segmentation : Rubinstein et al. , CVPR 2013 • Compositional Image Model : Mobahi et al. , CVPR 2014 • Object discovery and localization : Cho et al. , CVPR 2015 • FlowWeb : T. Zhou et al. , CVPR 2015 • Multi-image Matching : X. Zhou et al. , ICCV 2015 •

Labels for CNN Training? CNN Infeasible to label in large-scale

Cycle-consistency as Supervision • Composite flows along a cycle should be zero

Cycle-consistency as Supervision • Composite flows along a cycle should be zero • 2-cycle consistency: F i,j � F j,i = 0

Cycle-consistency as Supervision • Composite flows along a cycle should be zero • 2-cycle consistency: F i,j � F j,i = 0 • 3-cycle consistency: F i,k � F k,j � F j,i = 0

Cycle-consistency as Supervision • Composite flows along a cycle should be zero • 2-cycle consistency: F i,j � F j,i = 0 • 3-cycle consistency: F i,k � F k,j � F j,i = 0

Cycle-consistency as Supervision • Composite flows along a cycle should be zero • 2-cycle consistency: F i,j � F j,i = 0 • 3-cycle consistency: F i,k � F k,j � F j,i = 0 Amount of CNN inconsistency

Cycle Consistency in Vision Shape Matching Co-segmentation SfM Huang et al, SGP’13 Wang et al, ICCV’13 Zach et al , CVPR’10 Collection Correspondence Zhou et al, CVPR’15 Zhou et al, ICCV’15

Could be consistent but wrong …   0 0   0 0   0   . .  0  . . 0 . . . . . . 0 . 0 0 0 . . . 0 . . 0 . 0 . . . . . 0 . . . . 0 0 . . 0  .  . 0 0 0 . .  .  . 0   . . . . .  .  . . Need an anchor edge!   0 0   0 0 0 0 . . . 0 0 0 0   . . .    . . . .  . . . .   . . . . 0 0 0 0 . . .

Synthetic Correspondence as the Anchor Viewpoint Renderer Correspondence from renderer 3D CAD Model

3D-guided Cycle Consistency ˜ synthetic s 1 F s 1 ,s 2 synthetic s 2 Ground truth F s 1 ,r 1 F r 2 ,s 2 real r 1 real r 2 F r 1 ,r 2 Accumulate flow vector ˜ F s 1 ,s 2 = F s 1 ,r 1 � F r 1 ,r 2 � F r 2 ,s 2

3D-guided Cycle Consistency ˜ synthetic s 1 F s 1 ,s 2 synthetic s 2 Ground truth F s 1 ,r 1 F r 2 ,s 2 TRAINING TIME real r 1 real r 2 F r 1 ,r 2 ⇣ ⌘ X ˜ L F s 1 ,s 2 � F s 1 ,r 1 � F r 1 ,r 2 � F r 2 ,s 2 min < s 1 ,s 2 ,r 1 ,r 2 >

Network Architecture Source 3 16 32 32 64 64 128 128 256 8 16 16 32 32 64 64 Flow field 128 128 32 2 64 64 Weight 128 128 256 256 512 Sharing 8 16 16 32 32 64 64 128 128 Target 3 16 32 32 64 64 128 128 256 8 16 16 32 32 64 64 128 128

Matchability Prediction Source Flow field CNN Target

Matchability Prediction Source Flow field CNN Target Background: ✗ !

Matchability Prediction Source Flow field CNN Target Background: ✗ ! Occlusion: ✗ !

Matchability Prediction Source Matchability CNN Flow field Target

Training Set Construction PASCAL 3D ShapeNet (Bbox + Viewpoint) (Synthetic Rendering) Xiang et al , WACV’14 Chang et al , arXiv’15

Training Set Construction PASCAL 3D ShapeNet (Bbox + Viewpoint) (Synthetic Rendering) Xiang et al , WACV’14 Chang et al , arXiv’15

Training Set Construction Image-to-shape retrieval … … … … Single view reconstruction via joint analysis of image and shape collections , Huang et al ., SIGGRAPH 2015

Training Set Construction One training example • ~80,000 examples per category • A single network for all 12 PASCAL3D categories (aero, boat, bus, car, chair, etc.)

RESULTS

Image Warping Visualization Target Source SIFT flow Ours

Image Warping Visualization Source Target SIFT flow Ours

Keypoint Transfer Accuracy (PCK) Source Target SIFT Ours flow 19.6 24.0 Mean … 22.4 33.3 Car SIFT flow Ours 28.6 40.3 Bus 28.3 40.3 Bottle 42.9 51.1 TV …

Matchability Prediction Source Ours Ground truth Target Accuracy SIFT flow Ours 72.0 64.5

t-SNE Feature Visualization Global image features 3 16 Source 32 32 64 64 128 128 256 Flow field 32 2 64 64 128 128 256 256 512 8 16 16 32 32 64 64 128 128 8 16 16 32 32 64 64 128 128 Weight sharing Matchability 16 2 32 32 64 64 128 128 256 3 16 Target 32 32 64 64 128 128 256 8 16 16 32 32 64 64 128 128 8 16 16 32 32 64 64 128 128

t-SNE Feature Visualization 45 。 Side views views Frontal views

Application: Cross-domain Dense Label Transfer Ours Source Target Dense CRF SIFT flow

Conclusion ˜ synthetic s 1 F s 1 ,s 2 synthetic s 2 Ground truth F s 1 ,r 1 F r 2 ,s 2 TRAINING TIME real r 1 real r 2 F r 1 ,r 2 • Cycle consistency effective when direct labels not available • ‘Meta’-supervision : supervising the behavior of the data Thank you!