unsupervised learning of probably symmetric deformable 3d
play

Unsupervised Learning of Probably Symmetric Deformable 3D Objects - PowerPoint PPT Presentation

Aug 25, 2023 •2.87k likes •3.49k views

Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild Shangzhe Wu Christian Rupprecht Andrea Vedaldi V ISUAL G EOMETRY G ROUP , U NIVERSITY OF O XFORD Agenda v Problem Introduction v Method Overview v Results

  1. Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild Shangzhe Wu Christian Rupprecht Andrea Vedaldi V ISUAL G EOMETRY G ROUP , U NIVERSITY OF O XFORD

  2. Agenda v Problem Introduction v Method Overview v Results v Discussions v Conclusions 29

  3. What is 3D Reconstruction? Vision Reconstruction Rendering Graphics 2D Observations 3D Representation 30

  4. Multi-view 3D Reconstruction Building Rome in a Day. Agarwal et al. ICCV’09 static scene but.. the world is dynamic 33

  5. Multi-view 3D Reconstruction Building Rome in a Day. The Relightables: Volumetric Performance Capture of Humans Agarwal et al. ICCV’09 with Realistic Relighting. Guo et al. SIGGRAPH Asia’19 static scene 100 cameras too expensive for me :( 34

  6. Learning-based Single-view 3D Reconstruction Neural Network Need supervision ! 3D prior learned during training 35

  7. Supervision during Training 3D ground truth or multi-views depth maps shape models silhouettes keypoints camera viewpoint 36

  8. Unsupervised Learning of 3D Objects 3D ground truth or multi-views depth maps shape models silhouettes keypoints camera viewpoint 37

  9. Unsupervised Learning of 3D Objects Training Data Output Unsup3D single-view images of a category instance-specific 3D shapes NO other supervision! 38

  10. input reconstruction input reconstruction 39

  11. Unsupervised Learning of 3D Objects Training Data Output Unsup3D single-view images of a category instance-specific 3D shapes NO other supervision! 40

  12. Symmetries in the World 41

  13. Training Pipeline: Photo-Geometric Autoencoding 42

  14. Photo-Geometric Autoencoding input # encoder encoder encoder decoder decoder view ! depth " texture Reconstruction Loss Renderer reconstruction $ # 43

  15. Photo-Geometric Autoencoding Q1 : How to avoid degenerate solutions? input # encoder encoder encoder decoder decoder view ! depth " texture Reconstruction Loss Renderer reconstruction $ # 44

  16. Photo-Geometric Autoencoding Q1 : How to avoid degenerate solutions? A1 : Enforce symmetry input # encoder encoder encoder decoder decoder view ! depth " texture Reconstruction Loss Renderer reconstruction $ # 45

  17. Photo-Geometric Autoencoding Q1 : How to avoid degenerate solutions? A1 : Enforce symmetry by flipping : horizontal flip input # encoder encoder encoder decoder decoder ? ? view ! depth " depth "′ texture flipped 46

  18. Photo-Geometric Autoencoding Q1 : How to avoid degenerate solutions? A1 : Enforce symmetry by flipping : horizontal flip input # encoder encoder encoder decoder decoder ? ? view ! depth " depth "′ texture flipped flip switch Reconstruction Loss ? Renderer reconstruction $ # 47

  19. Photo-Geometric Autoencoding Q1 : How to avoid degenerate solutions? A1 : Enforce symmetry by flipping : horizontal flip input # encoder encoder encoder decoder decoder ? ? view ! depth " depth "′ texture flipped flip switch Reconstruction Loss Renderer reconstruction $ # 48

  20. Photo-Geometric Autoencoding Q1 : How to avoid degenerate solutions? A1 : Enforce symmetry by flipping : horizontal flip input # encoder encoder encoder decoder decoder ? ? view ! depth " depth "′ texture flipped flip switch Reconstruction Loss ? Renderer reconstruction $ # 49

  21. Photo-Geometric Autoencoding Q1 : How to avoid degenerate solutions? A1 : Enforce symmetry by flipping : horizontal flip input # encoder encoder encoder decoder decoder view ! depth " depth "′ texture flipped flip switch Reconstruction Loss Renderer reconstruction $ # 50

  22. Photo-Geometric Autoencoding Q2 : What about non-symmetric lighting? : horizontal flip input # encoder encoder encoder decoder decoder view ! depth " depth "′ texture flipped 51

  23. Photo-Geometric Autoencoding Q2 : What about non-symmetric lighting? A2 : Enforce symmetry on albedo : horizontal flip input # encoder encoder encoder encoder decoder decoder view ! depth " depth "′ light ' albedo ( albedo (′ flip switch Reconstruction Loss shading Renderer reconstruction $ canonical view & # 52

  24. Photo-Geometric Autoencoding Q3 : Non-symmetric albedo, deformation, etc? : horizontal flip input # encoder encoder encoder encoder decoder decoder view ! depth " depth "′ light ' albedo ( albedo (′ flip switch Reconstruction Loss shading Renderer reconstruction $ canonical view & # 53

  25. Photo-Geometric Autoencoding Q3 : Non-symmetric albedo, deformation, etc? A3 : Predict uncertainty : horizontal flip input # encoder encoder encoder encoder encoder decoder decoder decoder conf. ) conf. )′ view ! depth " depth "′ light ' albedo ( albedo (′ flip switch Reconstruction Loss shading Renderer reconstruction $ canonical view & # 54

  26. Photo-Geometric Autoencoding Q3 : Non-symmetric albedo, deformation, etc? A3 : Predict uncertainty : horizontal flip input # encoder encoder encoder encoder encoder decoder decoder decoder conf. ) conf. )′ view ! depth " depth "′ light ' albedo ( albedo (′ flip switch Reconstruction Loss shading Renderer reconstruction $ canonical view & # 55

  27. Photo-Geometric Autoencoding Q3 : Non-symmetric albedo, deformation, etc? A3 : Predict uncertainty : horizontal flip input # encoder encoder encoder encoder encoder decoder decoder decoder conf. ) conf. )′ view ! depth " depth "′ light ' albedo ( albedo (′ flip switch Reconstruction Loss shading Renderer reconstruction $ canonical view & # 56

  28. Photo-Geometric Autoencoding Q3 : Non-symmetric albedo, deformation, etc? A3 : Predict uncertainty : horizontal flip input # encoder encoder encoder encoder encoder decoder decoder decoder conf. ) conf. )′ view ! depth " depth "′ light ' albedo ( albedo (′ flip switch Reconstruction Loss shading Renderer reconstruction $ canonical view & # 57

  29. Photo-Geometric Autoencoding Q3 : Non-symmetric albedo, deformation, etc? A3 : Predict uncertainty : horizontal flip input # encoder encoder encoder encoder encoder decoder decoder decoder conf. ) conf. )′ view ! depth " depth "′ light ' albedo ( albedo (′ flip switch Reconstruction Loss shading Renderer reconstruction $ canonical view & # 58

  30. Photo-Geometric Autoencoding Q3 : Non-symmetric albedo, deformation, etc? A3 : Predict uncertainty : horizontal flip input # encoder encoder encoder encoder encoder decoder decoder decoder conf. ) conf. )′ view ! depth " depth "′ light ' albedo ( albedo (′ flip switch Reconstruction Loss shading Renderer reconstruction $ canonical view & # 59

  31. Photo-Geometric Autoencoding Q3 : Non-symmetric albedo, deformation, etc? A3 : Predict uncertainty : horizontal flip input # encoder encoder encoder encoder encoder decoder decoder decoder conf. ) conf. )′ view ! depth " depth "′ light ' albedo ( albedo (′ flip switch Reconstruction Loss shading Renderer reconstruction $ canonical view & # 60

  32. Results on human faces Images taken from CelebA, 3DFAW 61

  33. input reconstruction input reconstruction 62

  34. Results on face paintings Images taken from [1] 63 [1] Elliot J. Crowley, Omkar M. Parkhi, and Andrew Zisserman. Face painting: querying art with photos. In Proc. BMVC, 2015.

  35. input reconstruction input reconstruction 64

  36. Results on abstract faces Images taken from [1] and the Internet 65 [1] Elliot J. Crowley, Omkar M. Parkhi, and Andrew Zisserman. Face painting: querying art with photos. In Proc. BMVC, 2015.

  37. input reconstruction input reconstruction 66

  38. Results on video frames Video clips taken from VoxCeleb2 We do not use videos for training or fine-tuning. These results are obtained by applying our model trained on CelebA frame by frame . 67

  39. 68 recon. new view rotated recon. new view rotated input input recon. new view rotated recon. new view rotated input input

  40. Relighting effects Images taken from CelebA 69

  41. input reconstruction input reconstruction 70

  42. Results on cat faces Images taken from [2] and [3] [2] Weiwei Zhang, Jian Sun, and Xiaoou Tang. Cat head detection - how to effectively exploit shape and texture features. In Proc. ECCV, 2008. [3] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In Proc. CVPR, 2012. 71

  43. input reconstruction input reconstruction 72

  44. Results on synthetic cars Images rendered using ShapeNet 73

  45. input reconstruction input reconstruction 74

  46. Symmetry Plane Visualization 75

  47. Asymmetry Visualization 76

  48. Discussion: Ablation Studies 77

  49. Ablation – Symmetry full Input w/o albedo flip w/o depth flip Depth Normal Shading Albedo Shaded Recon. Insight #1 : Symmetry avoids degeneracy 78

  50. Ablation – Lighting (Shape from Shading) full Input w/o lighting Depth Normal Shading Albedo Shaded Recon. Insight #2 : Lighting avoids bumpy shapes and provides cues for shape 79

Recommend

UNSUPERVISED LEARNING, CLUSTERING UNSUPERVISED LEARNING UNSUPERVISED LEARNING Supervised

UNSUPERVISED LEARNING, CLUSTERING UNSUPERVISED LEARNING UNSUPERVISED LEARNING Supervised

UNSUPERVISED LEARNING, CLUSTERING UNSUPERVISED LEARNING UNSUPERVISED LEARNING Supervised learning: X - y pairs, f(x) function approximation Unsupervised learning: only X, no y Exploring the space of X measurements,

440 views • 14 slides
Introduction to PCA Unsupervised Learning in R Unsupervised learning Two methods of

Introduction to PCA Unsupervised Learning in R Unsupervised learning Two methods of

UNSUPERVISED LEARNING IN R Introduction to PCA Unsupervised Learning in R Unsupervised learning Two methods of clustering finding groups of homogeneous items Next up, dimensionality reduction Find structure in features

904 views • 29 slides
Unsupervised Learning and Clustering l In unsupervised learning you are given a data set with no

Unsupervised Learning and Clustering l In unsupervised learning you are given a data set with no

Unsupervised Learning and Clustering l In unsupervised learning you are given a data set with no output classifications (labels) l Clustering is an important type of unsupervised learning PCA was another type of unsupervised learning l The

858 views • 38 slides
Unsupervised Learning Unsupervised vs Supervised Learning: Most of this course focuses on

Unsupervised Learning Unsupervised vs Supervised Learning: Most of this course focuses on

Unsupervised Learning Unsupervised vs Supervised Learning: Most of this course focuses on supervised learning methods such as regression and classification. In that setting we observe both a set of features X 1 , X 2 , . . . , X p for each

934 views • 50 slides
Unsupervised Learning Unsupervised vs Supervised Learning: Most of this course focuses on

Unsupervised Learning Unsupervised vs Supervised Learning: Most of this course focuses on

Unsupervised Learning Unsupervised vs Supervised Learning: Most of this course focuses on supervised learning methods such as regression and classification. In that setting we observe both a set of features X 1 , X 2 , . . . , X p for each

721 views • 60 slides
Unsupervised Learning Andrea Passerini passerini@disi.unitn.it Machine Learning Unsupervised

Unsupervised Learning Andrea Passerini passerini@disi.unitn.it Machine Learning Unsupervised

Unsupervised Learning Andrea Passerini passerini@disi.unitn.it Machine Learning Unsupervised Learning Unsupervised Learning Setting Supervised learning requires the availability of labelled examples Labelling examples can be an extremely

752 views • 27 slides
12. Unsupervised Deep Learning CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from

12. Unsupervised Deep Learning CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from

12. Unsupervised Deep Learning CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from Wanli Ouyang, Zsolt Kira, Lawrence Neal, Raymond Yeh, Junting Lou and Teck-Yian Lim Unsupervised Learning in General Unsupervised learning is

1.48k views • 115 slides
Machine Learning for NLP Unsupervised Learning Aurlie Herbelot 2019 Centre for Mind/Brain

Machine Learning for NLP Unsupervised Learning Aurlie Herbelot 2019 Centre for Mind/Brain

Machine Learning for NLP Unsupervised Learning Aurlie Herbelot 2019 Centre for Mind/Brain Sciences University of Trento 1 Unsupervised learning In unsupervised learning, we learn without training data. The idea is to find a

998 views • 67 slides
Unsupervised Learning Introduction Nakul Verma Unsupervised Learning What can we learn from

Unsupervised Learning Introduction Nakul Verma Unsupervised Learning What can we learn from

Unsupervised Learning Introduction Nakul Verma Unsupervised Learning What can we learn from data when label information is not available? Supervised learning framework Supervised learning Data: Assumption: there is a (relatively simple)

442 views • 16 slides
Practical Unsupervised Learning INFO/CS 4300, Spring 2016 Jack Hessel Unsupervised Learning is

Practical Unsupervised Learning INFO/CS 4300, Spring 2016 Jack Hessel Unsupervised Learning is

Practical Unsupervised Learning INFO/CS 4300, Spring 2016 Jack Hessel Unsupervised Learning is Cool! But how can we use this in our projects? 1. 2. But first, lets look at our dataset... Data Dimensionality! Probability of Lung Cancer

999 views • 55 slides
Automatic Learning of a Morphological Model Theory and Unsupervised Approaches Unsupervised

Automatic Learning of a Morphological Model Theory and Unsupervised Approaches Unsupervised

Automatic Learning of a Morphological Model Theory and Unsupervised Approaches Unsupervised Learning - ToC Foundations Problem description General architecture Papers: Goldsmith 2001 and 2006 Goldsmith and Hu 2004

1.23k views • 43 slides
Unsupervised learning introduction October 7, 2019 Unsupervised learning introduction

Unsupervised learning introduction October 7, 2019 Unsupervised learning introduction

Unsupervised learning introduction October 7, 2019 Unsupervised learning introduction October 7, 2019 1 / 39 Intro General statement of the problem One has a set of N observations ( x 1 , x 2 , ..., x N ) of a random p -vector X having

495 views • 32 slides
Unsupervised learning General introduction to unsupervised learning PCA Special directions

Unsupervised learning General introduction to unsupervised learning PCA Special directions

Unsupervised learning General introduction to unsupervised learning PCA Special directions These are special directions we will try to find. Best direction u : |u| 2 = 1 2 1. Minimize : d i X i T u is the projection length x i u d i T

1.01k views • 29 slides
Learning to Select Expert Demonstra4ons for Deformable Object

Learning to Select Expert Demonstra4ons for Deformable Object

Learning to Select Expert Demonstra4ons for Deformable Object Manipula4on Dylan Hadfield-Menell, Alex Lee, Sandy Huang, Eric Tzeng, Pieter Abbeel Workshop on

438 views • 22 slides
THE MAGIC OF UNSUPERVISED LEARNING Agustinus Nalwan Head of AI Carsales.com.au A LITTLE BIT

THE MAGIC OF UNSUPERVISED LEARNING Agustinus Nalwan Head of AI Carsales.com.au A LITTLE BIT

THE MAGIC OF UNSUPERVISED LEARNING Agustinus Nalwan Head of AI Carsales.com.au A LITTLE BIT ABOUT MYSELF AI 4 Years SAVING OUR FOREST BUSHFIRE DETECTION BACK TO THE MAIN TOPIC THE MAGIC OF UNSUPERVISED LEARNING SUPERVISED LEARNING

1.11k views • 82 slides
Lecture 13: From Unsupervised to Reinforcement Learning (Chapters 8-10) R. Rao, 528: Lecture 13

Lecture 13: From Unsupervised to Reinforcement Learning (Chapters 8-10) R. Rao, 528: Lecture 13

CSE/NB 528 Lecture 13: From Unsupervised to Reinforcement Learning (Chapters 8-10) R. Rao, 528: Lecture 13 1 Todays Agenda: All about Learning F Unsupervised Learning Sparse Coding Predictive Coding F Supervised learning Perceptrons and

479 views • 24 slides
Geometric Registration for Deformable Shapes 2.2 Deformable Registration Variational Model

Geometric Registration for Deformable Shapes 2.2 Deformable Registration Variational Model

Geometric Registration for Deformable Shapes 2.2 Deformable Registration Variational Model Deformable ICP Variational Model What is deformable shape matching? Example ? What are the Correspondences? 3 Eurographics 2010 Course

319 views • 30 slides
Unsupervised Deep Learning Tutorial - Part 2 Alex Graves MarcAurelio Ranzato

Unsupervised Deep Learning Tutorial - Part 2 Alex Graves MarcAurelio Ranzato

Unsupervised Deep Learning Tutorial - Part 2 Alex Graves MarcAurelio Ranzato gravesa@google.com ranzato@fb.com NeurIPS, 3 December 2018 Overview Practical Recipes of Unsupervised Learning Learning representations Learning to

1.16k views • 103 slides
Unsupervised Learning About this class Build a model for your data. Which datapoints

Unsupervised Learning About this class Build a model for your data. Which datapoints

Unsupervised Learning About this class Build a model for your data. Which datapoints Unsupervised learning are similar? k -means Clustering Nowadays lots of work on using unlabeled data to improve the performance of supervised learn-

450 views • 6 slides
Detecting and Characterizing Events Unsupervised Machine Learning for Social Science Allison

Detecting and Characterizing Events Unsupervised Machine Learning for Social Science Allison

Detecting and Characterizing Events Unsupervised Machine Learning for Social Science Allison J.B. Chaney Princeton University supervised learning unsupervised learning methods input output prediction exploration goals 9 3 1 ? 5 6

920 views • 59 slides
lti 1 (typically) Unsupervised learning in NLP non-convex optimization lti 2

lti 1 (typically) Unsupervised learning in NLP non-convex optimization lti 2

Concavity and Initialization for Unsupervised Dependency Parsing Kevin Gimpel Noah A. Smith lti 1 (typically) Unsupervised learning in NLP non-convex optimization lti 2 Dependency Model with Valence (Klein & Manning,

718 views • 45 slides
Clustering ! Unsupervised learning: learning without a teacher ! Only features and no labels

Clustering ! Unsupervised learning: learning without a teacher ! Only features and no labels

So far in the course Supervised learning: learning with a teacher ! You had training data which was (feature, label) pairs and the goal was to learn a mapping from features to labels Clustering ! Unsupervised learning: learning without a

164 views • 12 slides
Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning

Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning

GCT634/AI613: Musical Applications of Machine Learning (Fall 2020) Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning Pipeline in Classification Tasks A set of hand-designed audio features are selected

398 views • 26 slides
Unsupervised Language Learning: Representation Learning for NLP Katia Shutova ILLC University

Unsupervised Language Learning: Representation Learning for NLP Katia Shutova ILLC University

Unsupervised Language Learning: Representation Learning for NLP Unsupervised Language Learning: Representation Learning for NLP Katia Shutova ILLC University of Amsterdam 3 April 2018 Unsupervised Language Learning: Representation Learning

898 views • 76 slides

More recommend