creativeai 3d geometric domain
play

CreativeAI 3D (Geometric) Domain Niloy Mitra Iasonas Kokkinos - PowerPoint PPT Presentation

CreativeAI 3D (Geometric) Domain Niloy Mitra Iasonas Kokkinos Paul Guerrero Nils Thuerey Tobias Ritschel UCL UCL UCL TUM UCL Timetable Niloy Paul Nils Introduction 2:15 pm X X X 2:25 pm X Machine Learning Basics Theory


  1. <latexit sha1_base64="63ak468AXCor9nBfBLOpfhlLeqM=">AB7XicbZDLSgMxFIbP1Fut6pLN8Ei1E2ZKQVdFt24s4K9QDuWTJpYzPJkGSEMvQd3LhQxK3v4863MW1noa0/BD7+cw45w9izrRx3W8nt7a+sbmV3y7s7O7tHxQPj1paJorQJpFcqk6ANeVM0KZhtNOrCiOAk7bwfh6Vm8/UaWZFPdmElM/wkPBQkawsVbrtiwequf9YsmtuHOhVfAyKEGmRr/41RtIkRUGMKx1l3PjY2fYmUY4XRa6CWaxpiM8ZB2LQocUe2n82n6Mw6AxRKZ8waO7+nkhxpPUkCmxnhM1IL9dm5n+1bmLCSz9lIk4MFWTxUZhwZCSanY4GTFi+MQCJorZXREZYWJsQEVbAje8smr0KpWPMt3tVL9KosjDydwCmXw4ALqcAMNaAKBR3iGV3hzpPivDsfi9ack80cwx85nz9g645U</latexit> <latexit sha1_base64="63ak468AXCor9nBfBLOpfhlLeqM=">AB7XicbZDLSgMxFIbP1Fut6pLN8Ei1E2ZKQVdFt24s4K9QDuWTJpYzPJkGSEMvQd3LhQxK3v4863MW1noa0/BD7+cw45w9izrRx3W8nt7a+sbmV3y7s7O7tHxQPj1paJorQJpFcqk6ANeVM0KZhtNOrCiOAk7bwfh6Vm8/UaWZFPdmElM/wkPBQkawsVbrtiwequf9YsmtuHOhVfAyKEGmRr/41RtIkRUGMKx1l3PjY2fYmUY4XRa6CWaxpiM8ZB2LQocUe2n82n6Mw6AxRKZ8waO7+nkhxpPUkCmxnhM1IL9dm5n+1bmLCSz9lIk4MFWTxUZhwZCSanY4GTFi+MQCJorZXREZYWJsQEVbAje8smr0KpWPMt3tVL9KosjDydwCmXw4ALqcAMNaAKBR3iGV3hzpPivDsfi9ack80cwx85nz9g645U</latexit> <latexit sha1_base64="63ak468AXCor9nBfBLOpfhlLeqM=">AB7XicbZDLSgMxFIbP1Fut6pLN8Ei1E2ZKQVdFt24s4K9QDuWTJpYzPJkGSEMvQd3LhQxK3v4863MW1noa0/BD7+cw45w9izrRx3W8nt7a+sbmV3y7s7O7tHxQPj1paJorQJpFcqk6ANeVM0KZhtNOrCiOAk7bwfh6Vm8/UaWZFPdmElM/wkPBQkawsVbrtiwequf9YsmtuHOhVfAyKEGmRr/41RtIkRUGMKx1l3PjY2fYmUY4XRa6CWaxpiM8ZB2LQocUe2n82n6Mw6AxRKZ8waO7+nkhxpPUkCmxnhM1IL9dm5n+1bmLCSz9lIk4MFWTxUZhwZCSanY4GTFi+MQCJorZXREZYWJsQEVbAje8smr0KpWPMt3tVL9KosjDydwCmXw4ALqcAMNaAKBR3iGV3hzpPivDsfi9ack80cwx85nz9g645U</latexit> <latexit sha1_base64="63ak468AXCor9nBfBLOpfhlLeqM=">AB7XicbZDLSgMxFIbP1Fut6pLN8Ei1E2ZKQVdFt24s4K9QDuWTJpYzPJkGSEMvQd3LhQxK3v4863MW1noa0/BD7+cw45w9izrRx3W8nt7a+sbmV3y7s7O7tHxQPj1paJorQJpFcqk6ANeVM0KZhtNOrCiOAk7bwfh6Vm8/UaWZFPdmElM/wkPBQkawsVbrtiwequf9YsmtuHOhVfAyKEGmRr/41RtIkRUGMKx1l3PjY2fYmUY4XRa6CWaxpiM8ZB2LQocUe2n82n6Mw6AxRKZ8waO7+nkhxpPUkCmxnhM1IL9dm5n+1bmLCSz9lIk4MFWTxUZhwZCSanY4GTFi+MQCJorZXREZYWJsQEVbAje8smr0KpWPMt3tVL9KosjDydwCmXw4ALqcAMNaAKBR3iGV3hzpPivDsfi9ack80cwx85nz9g645U</latexit> <latexit sha1_base64="pPG/+UG2ht75Yz5Ajs7KOd+Azk=">AB7XicbZDLSgMxFIbP1Fut6pLN8Ei1E2ZsYIui27cWcFeoB1LJs20sZlkSDJCGfoOblwo4tb3cefbmLaz0NYfAh/OYec8wcxZ9q47reTW1ldW9/Ibxa2tnd294r7B0tE0Vog0guVTvAmnImaMw2k7VhRHAaetYHQ9rbeqNJMinszjqkf4YFgISPYWKt5WxYP1dNeseRW3JnQMngZlCBTvVf86vYlSIqDOFY647nxsZPsTKMcDopdBNY0xGeEA7FgWOqPbT2bYTdGKdPgqlsk8YNHN/T6Q40nocBbYzwmaoF2tT879aJzHhpZ8yESeGCjL/KEw4MhJNT0d9pigxfGwBE8XsrogMscLE2IAKNgRv8eRlaJ5VPMt356XaVRZHo7gGMrgwQXU4Abq0ACj/AMr/DmSOfFeXc+5q05J5s5hD9yPn8AYnCOVQ=</latexit> <latexit sha1_base64="pPG/+UG2ht75Yz5Ajs7KOd+Azk=">AB7XicbZDLSgMxFIbP1Fut6pLN8Ei1E2ZsYIui27cWcFeoB1LJs20sZlkSDJCGfoOblwo4tb3cefbmLaz0NYfAh/OYec8wcxZ9q47reTW1ldW9/Ibxa2tnd294r7B0tE0Vog0guVTvAmnImaMw2k7VhRHAaetYHQ9rbeqNJMinszjqkf4YFgISPYWKt5WxYP1dNeseRW3JnQMngZlCBTvVf86vYlSIqDOFY647nxsZPsTKMcDopdBNY0xGeEA7FgWOqPbT2bYTdGKdPgqlsk8YNHN/T6Q40nocBbYzwmaoF2tT879aJzHhpZ8yESeGCjL/KEw4MhJNT0d9pigxfGwBE8XsrogMscLE2IAKNgRv8eRlaJ5VPMt356XaVRZHo7gGMrgwQXU4Abq0ACj/AMr/DmSOfFeXc+5q05J5s5hD9yPn8AYnCOVQ=</latexit> <latexit sha1_base64="pPG/+UG2ht75Yz5Ajs7KOd+Azk=">AB7XicbZDLSgMxFIbP1Fut6pLN8Ei1E2ZsYIui27cWcFeoB1LJs20sZlkSDJCGfoOblwo4tb3cefbmLaz0NYfAh/OYec8wcxZ9q47reTW1ldW9/Ibxa2tnd294r7B0tE0Vog0guVTvAmnImaMw2k7VhRHAaetYHQ9rbeqNJMinszjqkf4YFgISPYWKt5WxYP1dNeseRW3JnQMngZlCBTvVf86vYlSIqDOFY647nxsZPsTKMcDopdBNY0xGeEA7FgWOqPbT2bYTdGKdPgqlsk8YNHN/T6Q40nocBbYzwmaoF2tT879aJzHhpZ8yESeGCjL/KEw4MhJNT0d9pigxfGwBE8XsrogMscLE2IAKNgRv8eRlaJ5VPMt356XaVRZHo7gGMrgwQXU4Abq0ACj/AMr/DmSOfFeXc+5q05J5s5hD9yPn8AYnCOVQ=</latexit> <latexit sha1_base64="pPG/+UG2ht75Yz5Ajs7KOd+Azk=">AB7XicbZDLSgMxFIbP1Fut6pLN8Ei1E2ZsYIui27cWcFeoB1LJs20sZlkSDJCGfoOblwo4tb3cefbmLaz0NYfAh/OYec8wcxZ9q47reTW1ldW9/Ibxa2tnd294r7B0tE0Vog0guVTvAmnImaMw2k7VhRHAaetYHQ9rbeqNJMinszjqkf4YFgISPYWKt5WxYP1dNeseRW3JnQMngZlCBTvVf86vYlSIqDOFY647nxsZPsTKMcDopdBNY0xGeEA7FgWOqPbT2bYTdGKdPgqlsk8YNHN/T6Q40nocBbYzwmaoF2tT879aJzHhpZ8yESeGCjL/KEw4MhJNT0d9pigxfGwBE8XsrogMscLE2IAKNgRv8eRlaJ5VPMt356XaVRZHo7gGMrgwQXU4Abq0ACj/AMr/DmSOfFeXc+5q05J5s5hD9yPn8AYnCOVQ=</latexit> What’s Different in 3D? • Number of Voxels grows as versus occupied surface O ( n 3 ) O ( n 2 ) �7

  2. Data Representation .. Many Possibilities! points �8

  3. Data Representation .. Many Possibilities! points voxels cells patches �8

  4. Challenges �9

  5. Challenges 1. Representation 
 �9

  6. Challenges 1. Representation 
 2. Neighborhood information • who are the neighbouring elements • how are the elements ordered 
 �9

  7. Challenges 1. Representation 
 2. Neighborhood information • who are the neighbouring elements • how are the elements ordered 
 3. Extrinsic versus intrinsic representation 
 �9

  8. Challenges 1. Representation 
 2. Neighborhood information • who are the neighbouring elements • how are the elements ordered 
 3. Extrinsic versus intrinsic representation 
 4. Simplicity versus memory/runtime tradeoff �9

  9. Representation for 3D • Image-based 
 • Volumetric 
 • Surface-based 
 • Point-based �10

  10. Representation for 3D • Image-based 
 • Volumetric 
 • Surface-based 
 • Point-based �11

  11. Representation for 3D: Multi-view CNN [Kalogerakis et al. 2015] �12

  12. Representation for 3D: Multi-view CNN [Kalogerakis et al. 2015] �12

  13. Representation for 3D: Multi-view CNN [Kalogerakis et al. 2015] �12

  14. Representation for 3D: Multi-view CNN regular image analysis networks [Kalogerakis et al. 2015] �12

  15. 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation �13

  16. Integrating View Information �14

  17. Representation for 3D: Local Multi-view CNN [Huang et al. 2018] �15

  18. Representation for 3D: Local Multi-view CNN Segmentation Correspondence Feature matching Predicting semantic functions [Huang et al. 2018] �15

  19. Representation for 3D: Local Multi-view CNN Segmentation Correspondence Feature matching Predicting semantic functions [Huang et al. 2018] localized renderings for point-wise features �15

  20. Tangent Convolutions loses information due to occlusion [Tatarchenko et al. 2018] �16

  21. Tangent Convolutions loses information due to occlusion project to local patches 
 (contrast with PCPNet construction) [Tatarchenko et al. 2018] �16

  22. Tangent Convolutions loses information due to occlusion project to local patches 
 (contrast with PCPNet construction) [Tatarchenko et al. 2018] �16

  23. Dealing with Sparse Points �17

  24. Dealing with Sparse Points �18

  25. Improved Performance �19

  26. Representation for 3D • Image-based • PROS: directly use image networks, good performance • CONS: rendering is slow and memory-heavy, not very geometric • Volumetric 
 • Point-based 
 • Surface-based �20

  27. Representation for 3D • Image-based 
 • Volumetric 
 • Surface-based 
 • Point-based �21

  28. 3D CNNs : Direct Approach [Xiao et al. 2014] �22

  29. VoxNet [Maturana et al. 15] ▸ Binary occupancy, density grid, etc. rotational invariance �23

  30. Visualization of First Level Filters �24

  31. Visualization of First Level Filters �24

  32. Representation for 3D: Volumetric Deformation [Yumer and Mitra 2016] �25

  33. Representation for 3D: Volumetric Deformation [Yumer and Mitra 2016] �25

  34. Efficient Volumetric Datastructures [Wang et al. 2017] �26

  35. Data Structure and CNN Operations shuffled keys labels downsampling example (encode position in space) (parent label → child indices) ( “where there is an octant, there is CNN computation” ) faster neighbor access �27

  36. Data Structure and CNN Operations shuffled keys labels downsampling example (encode position in space) (parent label → child indices) ( “where there is an octant, there is CNN computation” ) faster neighbor access �27

  37. Data Structure and CNN Operations shuffled keys labels downsampling example (encode position in space) (parent label → child indices) ( “where there is an octant, there is CNN computation” ) faster neighbor access �27

  38. Efficient Volumetric Datastructures Encoder Decoder/generator Wang et al. 2017 only generate non-empty voxels [Hane et al. 2018] �28

  39. Efficient Volumetric Datastructures [Hane et al. 2018] �29

  40. Lower Memory Footprint �30

  41. Adaptive O-CNN [Wang et al. 2018] image to planar patch-based shapes

  42. First-order Patches OCNN Adaptive OCNN �32

  43. Field Probing Neural Networks for 3D Data [Li et al. 2016] �33

  44. Field Probing Neural Networks for 3D Data [Li et al. 2016] �33

  45. Spatial Probes �34

  46. Spatial Probes �34

  47. Spatial Probes �34

  48. Method Details �35

  49. Method Details �35

  50. Method Details �35

  51. Method Details �35

  52. Representation for 3D • Image-based 
 • Volumetric • PROS: adaptations of image networks • CONS: special layers for hierarchical datastructures, still too coarse 
 • Surface-based 
 • Point-based �36

  53. Representation for 3D • Image-based 
 • Volumetric 
 • Surface-based 
 • Point-based �37

  54. Local/Global Parameterizations �38

  55. Local/Global Parameterizations Geometry Image [Sinha et al. 2016] �38

  56. Local/Global Parameterizations Geometry Image Metric Alignment (GWCNN) [Sinha et al. 2016] [Ezuz et al. 2017] �38

  57. Shape Surfaces using Geometry Images �39

  58. Shape Surfaces using Geometry Images �39

  59. Shape Surfaces using Geometry Images �39

  60. Using Geodesic Patches: GCNN [Masci et al. 2015] �40

  61. <latexit sha1_base64="TquvtCTwYBxPgA56TL/7lpYdxY0=">ACO3icbZDLSxBEMZ7NA8z5rHRYy6FizBDRGYkEAkIknjwaIKrws6y1PTWuI09D7prxGXY/yuX/BO5eIlh0jINf07s4hPgoafnxfFdX1pZVWlqPoh7e0/Ojxk6crz/zV5y9evuq8XjuxZW0k9WSpS3OWoiWtCuqxYk1nlSHMU02n6cWnmX96ScaqsjmSUWDHM8LlSmJ7KRh50uQWIZDWAIwVUIH/bAT2ydD5uEx8S4ZaAwYLhLSQHpB20HpjQDw7cWBYGZmshsNON9qO5gX3IW6hK9o6Gna+J6NS1jkVLDVa24+jigcNGlZS09RPaksVygs8p7DAnOyg2Z+xQ2nTKCrDTuFQxz9f+JBnNrJ3nqOnPksb3rzcSHvH7N2e6gUVMxVysSirNXAJsyBhpAxJ1hMHKI1yfwU5RoOSXdy+CyG+e/J9ONnZjh1/ftfd/9jGsSLeiA0RiFi8F/viUByJnpDiq7gWv8SN98376f32/ixal7x2Zl3cKu/vP2AxqYM=</latexit> <latexit sha1_base64="TquvtCTwYBxPgA56TL/7lpYdxY0=">ACO3icbZDLSxBEMZ7NA8z5rHRYy6FizBDRGYkEAkIknjwaIKrws6y1PTWuI09D7prxGXY/yuX/BO5eIlh0jINf07s4hPgoafnxfFdX1pZVWlqPoh7e0/Ojxk6crz/zV5y9evuq8XjuxZW0k9WSpS3OWoiWtCuqxYk1nlSHMU02n6cWnmX96ScaqsjmSUWDHM8LlSmJ7KRh50uQWIZDWAIwVUIH/bAT2ydD5uEx8S4ZaAwYLhLSQHpB20HpjQDw7cWBYGZmshsNON9qO5gX3IW6hK9o6Gna+J6NS1jkVLDVa24+jigcNGlZS09RPaksVygs8p7DAnOyg2Z+xQ2nTKCrDTuFQxz9f+JBnNrJ3nqOnPksb3rzcSHvH7N2e6gUVMxVysSirNXAJsyBhpAxJ1hMHKI1yfwU5RoOSXdy+CyG+e/J9ONnZjh1/ftfd/9jGsSLeiA0RiFi8F/viUByJnpDiq7gWv8SN98376f32/ixal7x2Zl3cKu/vP2AxqYM=</latexit> <latexit sha1_base64="TquvtCTwYBxPgA56TL/7lpYdxY0=">ACO3icbZDLSxBEMZ7NA8z5rHRYy6FizBDRGYkEAkIknjwaIKrws6y1PTWuI09D7prxGXY/yuX/BO5eIlh0jINf07s4hPgoafnxfFdX1pZVWlqPoh7e0/Ojxk6crz/zV5y9evuq8XjuxZW0k9WSpS3OWoiWtCuqxYk1nlSHMU02n6cWnmX96ScaqsjmSUWDHM8LlSmJ7KRh50uQWIZDWAIwVUIH/bAT2ydD5uEx8S4ZaAwYLhLSQHpB20HpjQDw7cWBYGZmshsNON9qO5gX3IW6hK9o6Gna+J6NS1jkVLDVa24+jigcNGlZS09RPaksVygs8p7DAnOyg2Z+xQ2nTKCrDTuFQxz9f+JBnNrJ3nqOnPksb3rzcSHvH7N2e6gUVMxVysSirNXAJsyBhpAxJ1hMHKI1yfwU5RoOSXdy+CyG+e/J9ONnZjh1/ftfd/9jGsSLeiA0RiFi8F/viUByJnpDiq7gWv8SN98376f32/ixal7x2Zl3cKu/vP2AxqYM=</latexit> <latexit sha1_base64="TquvtCTwYBxPgA56TL/7lpYdxY0=">ACO3icbZDLSxBEMZ7NA8z5rHRYy6FizBDRGYkEAkIknjwaIKrws6y1PTWuI09D7prxGXY/yuX/BO5eIlh0jINf07s4hPgoafnxfFdX1pZVWlqPoh7e0/Ojxk6crz/zV5y9evuq8XjuxZW0k9WSpS3OWoiWtCuqxYk1nlSHMU02n6cWnmX96ScaqsjmSUWDHM8LlSmJ7KRh50uQWIZDWAIwVUIH/bAT2ydD5uEx8S4ZaAwYLhLSQHpB20HpjQDw7cWBYGZmshsNON9qO5gX3IW6hK9o6Gna+J6NS1jkVLDVa24+jigcNGlZS09RPaksVygs8p7DAnOyg2Z+xQ2nTKCrDTuFQxz9f+JBnNrJ3nqOnPksb3rzcSHvH7N2e6gUVMxVysSirNXAJsyBhpAxJ1hMHKI1yfwU5RoOSXdy+CyG+e/J9ONnZjh1/ftfd/9jGsSLeiA0RiFi8F/viUByJnpDiq7gWv8SN98376f32/ixal7x2Zl3cKu/vP2AxqYM=</latexit> Using Geodesic Patches: GCNN X ( f ? a )( x ) := a ( ✓ + ∆ ✓ , r )( D ( x ) f )( r, ✓ ) [Masci et al. 2015] θ ,r �40

  62. <latexit sha1_base64="TquvtCTwYBxPgA56TL/7lpYdxY0=">ACO3icbZDLSxBEMZ7NA8z5rHRYy6FizBDRGYkEAkIknjwaIKrws6y1PTWuI09D7prxGXY/yuX/BO5eIlh0jINf07s4hPgoafnxfFdX1pZVWlqPoh7e0/Ojxk6crz/zV5y9evuq8XjuxZW0k9WSpS3OWoiWtCuqxYk1nlSHMU02n6cWnmX96ScaqsjmSUWDHM8LlSmJ7KRh50uQWIZDWAIwVUIH/bAT2ydD5uEx8S4ZaAwYLhLSQHpB20HpjQDw7cWBYGZmshsNON9qO5gX3IW6hK9o6Gna+J6NS1jkVLDVa24+jigcNGlZS09RPaksVygs8p7DAnOyg2Z+xQ2nTKCrDTuFQxz9f+JBnNrJ3nqOnPksb3rzcSHvH7N2e6gUVMxVysSirNXAJsyBhpAxJ1hMHKI1yfwU5RoOSXdy+CyG+e/J9ONnZjh1/ftfd/9jGsSLeiA0RiFi8F/viUByJnpDiq7gWv8SN98376f32/ixal7x2Zl3cKu/vP2AxqYM=</latexit> <latexit sha1_base64="TquvtCTwYBxPgA56TL/7lpYdxY0=">ACO3icbZDLSxBEMZ7NA8z5rHRYy6FizBDRGYkEAkIknjwaIKrws6y1PTWuI09D7prxGXY/yuX/BO5eIlh0jINf07s4hPgoafnxfFdX1pZVWlqPoh7e0/Ojxk6crz/zV5y9evuq8XjuxZW0k9WSpS3OWoiWtCuqxYk1nlSHMU02n6cWnmX96ScaqsjmSUWDHM8LlSmJ7KRh50uQWIZDWAIwVUIH/bAT2ydD5uEx8S4ZaAwYLhLSQHpB20HpjQDw7cWBYGZmshsNON9qO5gX3IW6hK9o6Gna+J6NS1jkVLDVa24+jigcNGlZS09RPaksVygs8p7DAnOyg2Z+xQ2nTKCrDTuFQxz9f+JBnNrJ3nqOnPksb3rzcSHvH7N2e6gUVMxVysSirNXAJsyBhpAxJ1hMHKI1yfwU5RoOSXdy+CyG+e/J9ONnZjh1/ftfd/9jGsSLeiA0RiFi8F/viUByJnpDiq7gWv8SN98376f32/ixal7x2Zl3cKu/vP2AxqYM=</latexit> <latexit sha1_base64="TquvtCTwYBxPgA56TL/7lpYdxY0=">ACO3icbZDLSxBEMZ7NA8z5rHRYy6FizBDRGYkEAkIknjwaIKrws6y1PTWuI09D7prxGXY/yuX/BO5eIlh0jINf07s4hPgoafnxfFdX1pZVWlqPoh7e0/Ojxk6crz/zV5y9evuq8XjuxZW0k9WSpS3OWoiWtCuqxYk1nlSHMU02n6cWnmX96ScaqsjmSUWDHM8LlSmJ7KRh50uQWIZDWAIwVUIH/bAT2ydD5uEx8S4ZaAwYLhLSQHpB20HpjQDw7cWBYGZmshsNON9qO5gX3IW6hK9o6Gna+J6NS1jkVLDVa24+jigcNGlZS09RPaksVygs8p7DAnOyg2Z+xQ2nTKCrDTuFQxz9f+JBnNrJ3nqOnPksb3rzcSHvH7N2e6gUVMxVysSirNXAJsyBhpAxJ1hMHKI1yfwU5RoOSXdy+CyG+e/J9ONnZjh1/ftfd/9jGsSLeiA0RiFi8F/viUByJnpDiq7gWv8SN98376f32/ixal7x2Zl3cKu/vP2AxqYM=</latexit> <latexit sha1_base64="TquvtCTwYBxPgA56TL/7lpYdxY0=">ACO3icbZDLSxBEMZ7NA8z5rHRYy6FizBDRGYkEAkIknjwaIKrws6y1PTWuI09D7prxGXY/yuX/BO5eIlh0jINf07s4hPgoafnxfFdX1pZVWlqPoh7e0/Ojxk6crz/zV5y9evuq8XjuxZW0k9WSpS3OWoiWtCuqxYk1nlSHMU02n6cWnmX96ScaqsjmSUWDHM8LlSmJ7KRh50uQWIZDWAIwVUIH/bAT2ydD5uEx8S4ZaAwYLhLSQHpB20HpjQDw7cWBYGZmshsNON9qO5gX3IW6hK9o6Gna+J6NS1jkVLDVa24+jigcNGlZS09RPaksVygs8p7DAnOyg2Z+xQ2nTKCrDTuFQxz9f+JBnNrJ3nqOnPksb3rzcSHvH7N2e6gUVMxVysSirNXAJsyBhpAxJ1hMHKI1yfwU5RoOSXdy+CyG+e/J9ONnZjh1/ftfd/9jGsSLeiA0RiFi8F/viUByJnpDiq7gWv8SN98376f32/ixal7x2Zl3cKu/vP2AxqYM=</latexit> Using Geodesic Patches: GCNN X ( f ? a )( x ) := a ( ✓ + ∆ ✓ , r )( D ( x ) f )( r, ✓ ) [Masci et al. 2015] θ ,r �40

  63. <latexit sha1_base64="TquvtCTwYBxPgA56TL/7lpYdxY0=">ACO3icbZDLSxBEMZ7NA8z5rHRYy6FizBDRGYkEAkIknjwaIKrws6y1PTWuI09D7prxGXY/yuX/BO5eIlh0jINf07s4hPgoafnxfFdX1pZVWlqPoh7e0/Ojxk6crz/zV5y9evuq8XjuxZW0k9WSpS3OWoiWtCuqxYk1nlSHMU02n6cWnmX96ScaqsjmSUWDHM8LlSmJ7KRh50uQWIZDWAIwVUIH/bAT2ydD5uEx8S4ZaAwYLhLSQHpB20HpjQDw7cWBYGZmshsNON9qO5gX3IW6hK9o6Gna+J6NS1jkVLDVa24+jigcNGlZS09RPaksVygs8p7DAnOyg2Z+xQ2nTKCrDTuFQxz9f+JBnNrJ3nqOnPksb3rzcSHvH7N2e6gUVMxVysSirNXAJsyBhpAxJ1hMHKI1yfwU5RoOSXdy+CyG+e/J9ONnZjh1/ftfd/9jGsSLeiA0RiFi8F/viUByJnpDiq7gWv8SN98376f32/ixal7x2Zl3cKu/vP2AxqYM=</latexit> <latexit sha1_base64="TquvtCTwYBxPgA56TL/7lpYdxY0=">ACO3icbZDLSxBEMZ7NA8z5rHRYy6FizBDRGYkEAkIknjwaIKrws6y1PTWuI09D7prxGXY/yuX/BO5eIlh0jINf07s4hPgoafnxfFdX1pZVWlqPoh7e0/Ojxk6crz/zV5y9evuq8XjuxZW0k9WSpS3OWoiWtCuqxYk1nlSHMU02n6cWnmX96ScaqsjmSUWDHM8LlSmJ7KRh50uQWIZDWAIwVUIH/bAT2ydD5uEx8S4ZaAwYLhLSQHpB20HpjQDw7cWBYGZmshsNON9qO5gX3IW6hK9o6Gna+J6NS1jkVLDVa24+jigcNGlZS09RPaksVygs8p7DAnOyg2Z+xQ2nTKCrDTuFQxz9f+JBnNrJ3nqOnPksb3rzcSHvH7N2e6gUVMxVysSirNXAJsyBhpAxJ1hMHKI1yfwU5RoOSXdy+CyG+e/J9ONnZjh1/ftfd/9jGsSLeiA0RiFi8F/viUByJnpDiq7gWv8SN98376f32/ixal7x2Zl3cKu/vP2AxqYM=</latexit> <latexit sha1_base64="TquvtCTwYBxPgA56TL/7lpYdxY0=">ACO3icbZDLSxBEMZ7NA8z5rHRYy6FizBDRGYkEAkIknjwaIKrws6y1PTWuI09D7prxGXY/yuX/BO5eIlh0jINf07s4hPgoafnxfFdX1pZVWlqPoh7e0/Ojxk6crz/zV5y9evuq8XjuxZW0k9WSpS3OWoiWtCuqxYk1nlSHMU02n6cWnmX96ScaqsjmSUWDHM8LlSmJ7KRh50uQWIZDWAIwVUIH/bAT2ydD5uEx8S4ZaAwYLhLSQHpB20HpjQDw7cWBYGZmshsNON9qO5gX3IW6hK9o6Gna+J6NS1jkVLDVa24+jigcNGlZS09RPaksVygs8p7DAnOyg2Z+xQ2nTKCrDTuFQxz9f+JBnNrJ3nqOnPksb3rzcSHvH7N2e6gUVMxVysSirNXAJsyBhpAxJ1hMHKI1yfwU5RoOSXdy+CyG+e/J9ONnZjh1/ftfd/9jGsSLeiA0RiFi8F/viUByJnpDiq7gWv8SN98376f32/ixal7x2Zl3cKu/vP2AxqYM=</latexit> <latexit sha1_base64="TquvtCTwYBxPgA56TL/7lpYdxY0=">ACO3icbZDLSxBEMZ7NA8z5rHRYy6FizBDRGYkEAkIknjwaIKrws6y1PTWuI09D7prxGXY/yuX/BO5eIlh0jINf07s4hPgoafnxfFdX1pZVWlqPoh7e0/Ojxk6crz/zV5y9evuq8XjuxZW0k9WSpS3OWoiWtCuqxYk1nlSHMU02n6cWnmX96ScaqsjmSUWDHM8LlSmJ7KRh50uQWIZDWAIwVUIH/bAT2ydD5uEx8S4ZaAwYLhLSQHpB20HpjQDw7cWBYGZmshsNON9qO5gX3IW6hK9o6Gna+J6NS1jkVLDVa24+jigcNGlZS09RPaksVygs8p7DAnOyg2Z+xQ2nTKCrDTuFQxz9f+JBnNrJ3nqOnPksb3rzcSHvH7N2e6gUVMxVysSirNXAJsyBhpAxJ1hMHKI1yfwU5RoOSXdy+CyG+e/J9ONnZjh1/ftfd/9jGsSLeiA0RiFi8F/viUByJnpDiq7gWv8SN98376f32/ixal7x2Zl3cKu/vP2AxqYM=</latexit> Using Geodesic Patches: GCNN X ( f ? a )( x ) := a ( ✓ + ∆ ✓ , r )( D ( x ) f )( r, ✓ ) [Masci et al. 2015] θ ,r �40

  64. GCNN Architecture �41

  65. Handling Rotational Ambiguity �42

  66. Parameterization for Surface Analysis map 3D surface to 2D domain [Maron et al. 2017] �43

  67. Parameterization for Surface Analysis map 3D surface to 2D domain [Maron et al. 2017] �43

  68. Parameterization for Surface Analysis [Maron et al. 2017] �44

  69. Parameterization for Surface Analysis [Maron et al. 2017] �44

  70. Parameterization for Surface Analysis • Map 3D surface to 2D domain 
 • One such mapping: flat torus (seamless => translation-invariant) 
 • Many mappings exists: sample a few and average result 
 • Which functions to map? 
 XYZ, normals, curvature, … [Maron et al. 2017] �45

  71. Parameterization for Surface Analysis [Maron et al. 2017] �46

  72. Texture Transfer (Parameterization + Alignment) [Wang et al. 2016] �47

  73. AtlasNet for Surface Generation condition decoded points on 2D patches [Groueix et al. 2018] �48

  74. AtlasNet for Surface Generation Latent representation can be inferred from images or point clouds [Groueix et al. 2018] �49

  75. AtlasNet for Surface Generation Quad Mesh is generated by mapping a regular grid in 2D domain to 3D points Latent representation can be inferred from images or point clouds [Groueix et al. 2018] �50

  76. AtlasNet for Surface Generation texture coordinates come for free!! Latent representation can be inferred from images or point clouds �51

  77. Representation for 3D • Image-based 
 • Volumetric 
 • Surface-based • PROS: parameterize + image networks (instrinsic representation) • CONS: suffers from parameterisation artefacts (local versus global distortion), 
 requires good quality mesh 
 • Point-based �52

  78. Representation for 3D • Image-based 
 • Volumetric 
 • Surface-based 
 • Point-based �53

  79. Representation for 3D: Point-based • Common representation: native representation 
 • Easy to obtain from meshes, depth scans, laser scans �54

  80. In Original Representation • Common representation 
 • Easy to obtain from meshes, depth scans, laser scans 
 • Unstructured (e.g., any permutation of points gives same shape!) [Qi et al. 2017] �55

  81. PointNet for Point Cloud Analysis permutation-invariant functions [Qi et al. 2017] �56

  82. PointNet for Point Cloud Analysis Use MLPs (h) and max-pooling (g) as simple symmetric functions [Qi et al. 2017] �57

  83. PointNet Architecture [Qi et al. 2017] �58

  84. PointNet for Point Cloud Analysis �59

  85. PointNet++ [Qi et al. 2018] �60

  86. PCPNet for Local Point Cloud Analysis [Guerrero et al. 2018] �61

Recommend


More recommend