Dynamic Graph CNN for learning on point clouds Wang Yue, et al. - PowerPoint PPT Presentation

Dynamic Graph CNN for learning on point clouds Wang Yue, et al. Otakar Jašek March 25, 2019 Otakar Jašek Dynamic Graph CNN for learning on point clouds

Point cloud learning – history ▸ Hand-crafted features ▸ View-based methods ▸ Voxel representation ▸ PointNet ▸ Geometric deep learning Otakar Jašek Dynamic Graph CNN for learning on point clouds

Point cloud characteristics ▸ Unordered set of points ▸ Varying number of points ▸ Invariant to rigid transformations ▸ Often no features, location only Otakar Jašek Dynamic Graph CNN for learning on point clouds

What to do To combat first two points, we need a function, that is: ▸ invariant to permutations ▸ capable of taking any number of parameters 1 Jaderberg, Max, Karen Simonyan, and Andrew Zisserman. "Spatial transformer networks." Advances in neural information processing systems. 2015. 2 Qi, Charles R., et al. "Pointnet: Deep learning on point sets for 3d classification and segmentation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. Otakar Jašek Dynamic Graph CNN for learning on point clouds

What to do To combat first two points, we need a function, that is: ▸ invariant to permutations ▸ capable of taking any number of parameters For example ∑ or max Spatial transformer network 1 should solve rigid transformation invariance 1 Jaderberg, Max, Karen Simonyan, and Andrew Zisserman. "Spatial transformer networks." Advances in neural information processing systems. 2015. 2 Qi, Charles R., et al. "Pointnet: Deep learning on point sets for 3d classification and segmentation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. Otakar Jašek Dynamic Graph CNN for learning on point clouds

What to do To combat first two points, we need a function, that is: ▸ invariant to permutations ▸ capable of taking any number of parameters For example ∑ or max Spatial transformer network 1 should solve rigid transformation invariance And if we combine these two important ingredients together, we get a PointNet 2 1 Jaderberg, Max, Karen Simonyan, and Andrew Zisserman. "Spatial transformer networks." Advances in neural information processing systems. 2015. 2 Qi, Charles R., et al. "Pointnet: Deep learning on point sets for 3d classification and segmentation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. Otakar Jašek Dynamic Graph CNN for learning on point clouds

PointNet Classification Network input mlp (64,64) mlp (64,128,1024) feature max mlp transform input points transform pool (512,256,k) 1024 nx64 nx64 nx3 nx3 nx1024 shared shared global feature k output scores point features output scores 64x64 3x3 T-Net T-Net transform transform nx128 nxm n x 1088 shared shared matrix matrix multiply multiply mlp (512,256,128) mlp (128,m) Segmentation Network ▸ It does not capture neighborhood information Otakar Jašek Dynamic Graph CNN for learning on point clouds

PointNet++ ▸ Sample n points at coarse scale, as far from each other as possible ▸ Generate "global" features on local neighborhood of selected points by PointNet ▸ Concatenate these features and repeat for finer scale and larger n Otakar Jašek Dynamic Graph CNN for learning on point clouds

PointNet++ ▸ Sample n points at coarse scale, as far from each other as possible ▸ Generate "global" features on local neighborhood of selected points by PointNet ▸ Concatenate these features and repeat for finer scale and larger n ▸ Really slow due to sampling of neighborhood and running multiple PointNets Otakar Jašek Dynamic Graph CNN for learning on point clouds

EdgeConv Generalization of convolution on graph edge_conv ( x i ) = j ∶( j , i )∈ ε θ j h ( x i , x j ) ◻ where ▸ ε is a set of edges of the graph ▸ θ j are parameters of the EdgeConv layer ▸ ◻ is aggregation function (usually ∑ , max ) ▸ h ( x i , x j ) is a feature function Otakar Jašek Dynamic Graph CNN for learning on point clouds

EdgeConv continued ▸ Resulting operation largely depends on choice of ◻ and h ( ⋅ , ⋅ ) ▸ ◻ = ∑ , h ( x i , x j ) = x j is classical convolution (sum over weighted neighbors) ▸ h ( x i , x j ) = x i is PointNet-style convolution. It captures only global information ▸ h ( x i , x j ) = x j − x i is another reasonable choice, encoding locality (and disregarding global information) ▸ Authors of the papers decided, that h ( x i , x j ) = ( x i , x j − x i ) , ◻ = max Otakar Jašek Dynamic Graph CNN for learning on point clouds

Dynamic graph computation ▸ Second key idea is to recompute graph using kNN after every EdgeConv ▸ First EdgeConv is operating on spatially near points, while subsequent on semantically near points. Otakar Jašek Dynamic Graph CNN for learning on point clouds

Full architecture Otakar Jašek Dynamic Graph CNN for learning on point clouds

Classification results CENT DYN XFORM M EAN C LASS A CCURACY (%) O VERALL A CCURACY (%) 88.8 91.2 X 88.8 91.5 X X 89.6 91.9 X X 89.8 91.9 X X 90.2 92.2 X X X N UMBER OF NEAREST NEIGHBORS ( K ) M EAN O VERALL C LASS A CCURACY (%) A CCURACY (%) 5 88.0 90.5 10 88.8 91.4 20 90.2 92.2 40 89.2 91.7 Otakar Jašek Dynamic Graph CNN for learning on point clouds

Classification results continued M EAN O VERALL C LASS A CCURACY A CCURACY 3DS HAPE N ETS [54] 77.3 84.7 V OX N ET [30] 83.0 85.9 S UBVOLUME [35] 86.0 89.2 ECC [45] 83.2 87.4 P OINT N ET [34] 86.0 89.2 P OINT N ET ++ [36] - 90.7 K D -N ET ( DEPTH 10) [20] - 90.6 K D -N ET ( DEPTH 15) [20] - 91.8 O URS ( BASELINE ) 88.8 91.2 O URS 90.2 92.2 Otakar Jašek Dynamic Graph CNN for learning on point clouds

Segmentation results M EAN OVERALL I O U ACCURACY P OINT N ET (B ASELINE ) [34] 20.1 53.2 P OINT N ET [34] 47.6 78.5 MS + CU(2) [12] 47.8 79.2 G + RCU [12] 49.7 81.1 O URS 56.1 84.1 PointNet Ours Ground truth Real color Otakar Jašek Dynamic Graph CNN for learning on point clouds

Part segmentation PointNet Ours Ground truth Otakar Jašek Dynamic Graph CNN for learning on point clouds

Semantic nearness Otakar Jašek Dynamic Graph CNN for learning on point clouds

Semantic nearness across objects Source points Other point clouds from the same category Figure 9. Visualize the Euclidean distance (yellow: near, blue: far) Otakar Jašek Dynamic Graph CNN for learning on point clouds

Generating normals Otakar Jašek Dynamic Graph CNN for learning on point clouds