Disentangled Graph Convolutional Networks Jianxin Ma, Peng Cui, Kun - - PowerPoint PPT Presentation

Disentangled GCNs (ICML’19)

Disentangled Graph Convolutional Networks

Jianxin Ma, Peng Cui, Kun Kuang, Xin Wang, Wenwu Zhu Tsinghua University

Disentangled GCNs (ICML’19)

Motivation

• The neighborhood of a node is formed due to many latent factors.
• Existing GCNs convolute the neighborhood as a whole.
• They do not distinguish between the latent factors.
• Their node representations are thus not robust, and hardly interpretable.

𝑣

𝑤# 𝑤$ 𝑤% 𝑤

&

𝑤' 𝑤( 𝑤) 𝑤*

𝑣

𝑤# 𝑤$ 𝑤%

Latent factor: Work

𝑣

𝑤) 𝑤*

Latent factor: Family

𝑣

𝑤

&

𝑤' 𝑤(

Latent factor: Hobby

Disentangled GCNs (ICML’19)

Disentangled GCNs

• Disentangled representation learning aims to identify and separate the underlying

explanatory factors behind the observed data (Bengio et al., 2013).

• We identify the latent factors, and segment the neighborhood accordingly.
• Each segment is related with an isolated factor, and is convoluted separately.

𝑣

𝑤# 𝑤$ 𝑤% 𝑤

&

𝑤' 𝑤( 𝑤) 𝑤*

Neighborhood Routing Extract features specific to each factor.

convolution convolution convolution

𝑣

concatenate

Layer Output

𝑤) 𝑤*

𝑿𝟑

𝑤) 𝑤* 𝑤% 𝑤# 𝑤$ 𝑤% 𝑤# 𝑤$

𝑿𝟐

𝑤& 𝑤( 𝑤' 𝑤& 𝑤( 𝑤'

𝑿𝟒

Layer Input 𝑣

𝑤# 𝑤$ 𝑤% 𝑤

&

𝑤' 𝑤( 𝑤) 𝑤*

Feed back to improve neighborhood routing.

Disentangled GCNs (ICML’19)

Neighborhood Routing

• We propose neighborhood routing, to segment a neighborhood.
• Dynamic & differentiable. Similar to capsule networks’ dynamic routing.
• Phase I:
• To extract factor-specific features.

§ For node 𝑗 ∈ 𝑣 ∪ 𝑤: 𝑤, 𝑣 ∈ 𝐻 , and factor 𝑙 ∈ 1,2, … , 𝐿 , § 𝒜;,< =

>(𝑿@

A 𝒚CD𝒄@)

>(𝑿@

A 𝒚CD𝒄@) G

§ which describes node 𝑗’s aspect 𝑙.

• Phase II:
• To infer the factor that causes the link

between node 𝑣 and a neighbor 𝑤. § Initialize 𝒅< ← 𝒜J,< for each factor 𝑙. § Iterate for 𝑈 ≈ 5 times, § 𝑞O,< ←

PQR 𝒜S,@

A

𝒅@ /U ∑@W PQR 𝒜S,@W

A

𝒅@W /U

§ 𝒅< ←

𝒜X,@D∑S: S,X ∈Y ZS,@ 𝒜S,@ 𝒜X,@D∑S: S,X ∈Y ZS,@ 𝒜S,@

G

§ 𝒅< describes the neighborhood’s aspect 𝑙.

Disentangled GCNs (ICML’19)

Intuitions & Theories

• The two intuitions behind neighborhood routing:
• 𝑞 Factor 𝑙 is the one that causes the links between node 𝑣 and a segment ∝

The segment contains a large number of nodes that are similar w.r.t. aspect 𝑙.

• 𝑞 Factor 𝑙 is the one that causes the link between node 𝑣 and a neighbor ∝

Node 𝑣 and the neighbor are similar w.r.t. aspect 𝑙.

• Neighborhood routing is equivalent to an EM algorithm that performs

inference under a von Mises-Fisher subspace clustering model.

• It finds one large cluster in each of the 𝐿 subspaces.

Disentangled GCNs (ICML’19)

Results: Multi-label Node Classification

Disentangled GCNs (ICML’19)

Results: Disentangled Node Representations

• Correlations between the 64 dimensions, on a graph with eight factors.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1