LatentGNN: Learning Efficient Non-local Relations for Visual - PowerPoint PPT Presentation

LatentGNN: Learning Efficient Non-local Relations for Visual Recognition Songyang Zhang, Shipeng Yan, Xuming He ShanghaiTech University Songyang Zhang sy.zhangbuaa@gmail.com June 13, 2019

Goal & Motivation 1 Goal Learning efficient feature augmentation with Non-local relations for visual recognitions. Motivation ◮ To model the non-local feature context by a Graph Neural Network (GNN) . ◮ Self-attention Mechanism, Non-local network as special examples of Graph Neural Network with truncated inference. ◮ To reduce the complexity of a fully-connected GNN by introducing a latent representation . Dual Attention Network(Fu et al) Non-local Network(Wang et al) Attention is All You Need(Vaswani et al) |

Non-local Features with GNN 2 Notation x i x i ˜ ◮ Input : Grid/Non-grid Conv-feature, X = [ x 1 , · · · , x N ] T , x i ∈ R c ◮ Output : Context-aware Conv-feature, ˜ x N ] T , ˜ X = [˜ x 1 , · · · , ˜ x i ∈ R c ◮ Each Location :   N 1 Non-local features with GNN � g ( x i , x j ) W ⊤ x j x i = h ˜ (1)   Z i ( X ) j =1 ◮ Matrix Form : ˜ X aug = λ · ˜ X = h ( A ( X ) XW ) , X + X (2) ◮ g ( x i , x j ) = x T i x j : Pair-wise relations function ◮ h : Element-wise activation function(ReLU) ◮ Z i ( X ) : Normalization factor ◮ W ∈ R c × c : Weight matrix of the linear mapping If N = 500 × 500 , A requires 500GB of ◮ λ : Scaling parameter storage!!! |

Latent Graph Neural Network 3 LatentGNN ◮ Key Idea : Introduce a latent space for efficient global context encoding ◮ Conv-feature Space: X = [ x 1 , · · · , x N ] T , x i ∈ R c ◮ Latent Space: Z = [ z 1 , · · · , z d ] T , z i ∈ R c , d ≪ N |

Latent Graph Neural Network 4 Step-1: Visible-to-Latent Propagation(Bipartite Graph) ◮ Each Latent Node: N 1 � m k ( X ) ψ ( x j , θ k ) W T x j , z k = 1 ≤ k ≤ d (3) j =1 ◮ Matrix Form : Z = Ψ ( X ) T XW (4) ψ ( x i ) = [ ψ ( x i , θ 1 ) m 1 ( X ) , · · · , ψ ( x i , θ d ) Ψ ( X ) = [ ψ ( x 1 ) , · · · , ψ ( x N )] T ∈ R N × d , m d ( X ) ] T (5) ◮ ψ ( x j , θ k ) : : encode the affinity between node x j and node z k ◮ m k ( X ) : the normalization factor |

Latent Graph Neural Network 5 Step-2: Latent-to-Latent Propagation(Fully-connected Graph) ◮ Each Latent Node: d � z k = f ( φ k , φ j , X ) z j , 1 ≤ k ≤ d (6) ˜ j =1 ◮ Matrix Form : F X = [ f ( φ i , φ j , X )] d × d (7) ˜ Z = F X Z (8) ◮ f ( φ k , φ j , X ) : data-dependent pair-wise relations between two latent nodes |

Latent Graph Neural Network 6 Step-3: Latent-to-Visible Propagation(Bipartite Graph) � d ◮ Each Visible Node: � � x i = h ψ ( x i , θ k ) ˜ 1 ≤ i ≤ N (9) ˜ z k , k =1 � � ◮ Matrix Form : ˜ Ψ ( X ) ˜ X = h (10) Z |

LatentGNN vs. GNN 7 Overall Process LatentGNN GNN ◮ ˜ ◮ ˜ � Ψ ( X ) F X Ψ ( X ) T XW � X = h X = h ( A ( X ) XW ) ◮ X aug = λ · ˜ ◮ X aug = λ · ˜ X + X X + X 1 ◮ A i,j = Z i ( X ) g ( x i , x j ) , A ( X ) ∈ R N × N ◮ A ( X ) = Ψ ( X ) F X Ψ ( X ) T O ( N · d ) O ( N · N ) |

Experimental Results 8 Grid Data: Object Detection/Instance Segmentation on MSCOCO ◮ +NLBlock : insert the non-local block in the last stage of the backbone. ◮ +LatentGNN : Integrate LatentGNN with the backbone at different stages. |

Experimental Results 9 Grid Data: Ablation Study on MSCOCO ◮ Effects of different backbone networks. ◮ A mixture of low-rank matrices. Non-grid Data: Point Cloud Semantic Segmentation on ScanNet |

Take Home Message 10 LatentGNN ◮ A novel graph neural network for efficient non-local relations learning. ◮ Introduce a latent space for efficient message propagation ◮ Our model has a modularized design, which can be easily incorporated into any layer in deep ConvNet Poster : Thu, Jun 13, 2019 Pacific Ballroom #28 Paper Code(available soon) |

Thanks!