NetGAN without GAN: From Random Walks to Low-Rank Approximations - PowerPoint PPT Presentation

NetGAN without GAN: From Random Walks to Low-Rank Approximations Luca Rendsburg, Holger Heidrich, Ulrike von Luxburg ICML 2020

1 3 Input 1 8 1 4 1 1 5 6 9 1 2 1 7 1 9 } 3 1 } NetGAN: Generating Graphs via Random Walks 1 8 Random walks 1 ,..., 1 5 5 1 6 1 2 1 GAN NetGAN 5 } Synthetic 3 1 } 4 1 1 4 6 ,..., 1 random walks 9 1 7 1 2 1 Edge probability matrix 3 1 3 1 3 1 Output 1 8 1 8 1 8 4 1 4 1 4 1 1 1 1 5 1 6 1 5 6 1 5 6 1 / 14 1 9 1 9 9 1 1 2 1 2 2 1 1 7 7 1 7 1

1 3 Input 1 8 1 4 1 1 5 6 1 9 2 1 1 7 9 } 1 3 } 8 1 Random walks 1 ,..., 1 5 5 1 6 1 2 1 GAN 5 } Synthetic 3 1 } 4 1 1 4 6 ,..., 1 random walks 1 9 7 1 2 1 Edge probability matrix 3 1 3 1 1 3 Output 1 8 1 8 8 1 1 4 4 1 4 1 1 1 1 5 1 6 1 5 6 1 5 6 1 / 14 1 9 1 9 9 1 1 2 1 2 2 1 1 7 7 1 7 1

1 3 Input 1 8 1 4 1 1 5 6 1 9 2 1 1 7 9 } 1 3 } 8 1 Random walks 1 ,..., 1 5 5 1 6 1 2 1 GAN 5 } Synthetic 3 1 } 4 1 1 4 6 ,..., 1 random walks 1 9 7 1 2 1 Edge probability matrix 3 1 3 1 1 3 Output 1 8 1 8 8 1 1 4 4 1 4 1 1 1 1 5 1 6 1 5 6 1 5 6 1 / 14 1 9 1 9 9 1 1 2 1 2 2 1 1 7 7 1 7 1

1 3 Input 1 8 1 4 1 1 5 6 9 1 2 1 7 1 9 } 3 1 } 1 8 Random walks 1 ,..., 1 5 5 1 6 1 1 2 Learn random GAN walk distribution 5 } Synthetic 3 1 } 1 4 1 4 6 ,..., 1 random walks 1 9 7 1 1 2 Edge probability matrix 1 3 1 3 3 1 Output 8 1 8 1 1 8 4 1 4 1 4 1 1 1 1 5 1 6 1 5 6 1 5 6 1 / 14 1 9 1 9 9 1 1 2 1 2 2 1 1 7 7 1 7 1

1 3 Input 1 8 1 4 1 1 5 6 9 1 2 1 7 1 9 } 3 1 } 1 8 Random walks 1 ,..., 1 5 5 1 6 1 1 2 Learn random GAN walk distribution 5 } Synthetic 3 1 } 1 4 1 4 6 ,..., 1 random walks 1 9 7 1 1 2 Edge probability matrix 1 3 1 3 3 1 Output 8 1 8 1 1 8 4 1 4 1 4 1 1 1 1 5 1 6 1 5 6 1 5 6 1 / 14 1 9 1 9 9 1 1 2 1 2 2 1 1 7 7 1 7 1

1 3 Input 1 8 1 4 1 1 5 6 9 1 2 1 7 1 9 } 3 1 } 1 8 Random walks 1 ,..., 1 5 5 1 6 1 1 2 Learn random GAN walk distribution 5 } Synthetic 3 1 } 1 4 1 4 6 ,..., 1 random walks 1 9 7 1 1 2 Edge probability matrix 1 3 1 3 3 1 Output 8 1 8 1 1 8 4 1 4 1 4 1 1 1 1 5 1 6 1 5 6 1 5 6 1 / 14 1 9 1 9 9 1 1 2 1 2 2 1 1 7 7 1 7 1

Our contribution Conceptual analysis 2 / 14

Our contribution Conceptual analysis § Inductive bias of NetGAN 2 / 14

Our contribution Conceptual analysis § Inductive bias of NetGAN § Bypass sampling random walks 2 / 14

Our contribution Conceptual analysis § Inductive bias of NetGAN § Bypass sampling random walks Simplified version (no GAN, no sampling): “Cross-Entropy Low-rank Logits (CELL)” 2 / 14

Our contribution Conceptual analysis § Inductive bias of NetGAN § Bypass sampling random walks Simplified version (no GAN, no sampling): “Cross-Entropy Low-rank Logits (CELL)” § Higher transparency 2 / 14

Our contribution Conceptual analysis § Inductive bias of NetGAN § Bypass sampling random walks Simplified version (no GAN, no sampling): “Cross-Entropy Low-rank Logits (CELL)” § Higher transparency § Comparable generalization performance 2 / 14

Our contribution Conceptual analysis § Inductive bias of NetGAN § Bypass sampling random walks Simplified version (no GAN, no sampling): “Cross-Entropy Low-rank Logits (CELL)” § Higher transparency § Comparable generalization performance § Huge speedup 2 / 14

NetGAN: learning step § Goal: learn random walk distribution NetGAN Generator Figure from Bojchevski et al. [2018] v 1 v 1 v T architecture G(z) architecture v 2 sample sample p 1 p N p 1 p N Generator W up W up z Discrimi- o 1 o T nator C 0 C 1 ⋯ Graph h 0 h 1 ⋯ D real D fake W down ⋯ Random z ⇠ N (0,I d ) walk (a) Generator architecture (b) NetGAN architecture 3 / 14

NetGAN: learning step § Goal: learn random walk distribution § Training set: unbiased random walks tp v p i q 0 , . . . , v p i q T qu i of length T over input graph NetGAN Generator Figure from Bojchevski et al. [2018] v 1 v 1 v T architecture G(z) architecture v 2 sample sample p 1 p N p 1 p N Generator W up W up z Discrimi- o 1 o T nator C 0 C 1 ⋯ Graph h 0 h 1 ⋯ D real D fake W down ⋯ Random z ⇠ N (0,I d ) walk (a) Generator architecture (b) NetGAN architecture 3 / 14

NetGAN: learning step § Goal: learn random walk distribution § Training set: unbiased random walks tp v p i q 0 , . . . , v p i q T qu i of length T over input graph § Generator: generate sequences tp w p i q 0 , . . . , w p i q T qu i of “synthetic” random walks NetGAN Generator Figure from Bojchevski et al. [2018] v 1 v 1 v T architecture G(z) architecture v 2 sample sample p 1 p N p 1 p N Generator W up W up z Discrimi- o 1 o T nator C 0 C 1 ⋯ Graph h 0 h 1 ⋯ D real D fake W down ⋯ Random z ⇠ N (0,I d ) walk (a) Generator architecture (b) NetGAN architecture 3 / 14

NetGAN: learning step § Goal: learn random walk distribution § Training set: unbiased random walks tp v p i q 0 , . . . , v p i q T qu i of length T over input graph § Generator: generate sequences tp w p i q 0 , . . . , w p i q T qu i of “synthetic” random walks § Discriminator: distinguish synthetic from real random walks NetGAN Generator Figure from Bojchevski et al. [2018] v 1 v 1 v T architecture G(z) architecture v 2 sample sample p 1 p N p 1 p N Generator W up W up z Discrimi- o 1 o T nator C 0 C 1 ⋯ Graph h 0 h 1 ⋯ D real D fake W down ⋯ Random z ⇠ N (0,I d ) walk (a) Generator architecture (b) NetGAN architecture 3 / 14

NetGAN: learning step § Goal: learn random walk distribution § Training set: unbiased random walks tp v p i q 0 , . . . , v p i q T qu i of length T over input graph § Generator: generate sequences tp w p i q 0 , . . . , w p i q T qu i of “synthetic” random walks § Discriminator: distinguish synthetic from real random walks § Architecture: LSTMs with Wasserstein loss NetGAN Generator Figure from Bojchevski et al. [2018] v 1 v 1 v T architecture G(z) architecture v 2 sample sample p 1 p N p 1 p N Generator W up W up z Discrimi- o 1 o T nator C 0 C 1 ⋯ Graph h 0 h 1 ⋯ D real D fake W down ⋯ Random z ⇠ N (0,I d ) walk (a) Generator architecture (b) NetGAN architecture 3 / 14

NetGAN: reconstruction step Generator 5 } 3 1 Generate many synthetic } 1 4 4 1 6 ,..., 1 random walks 9 1 1 7 1 2 Count transitions in score matrix S Transform S into edge probability matrix Sample edges 1 3 1 8 4 1 1 without replacement 5 1 6 9 1 1 2 4 / 14 1 7

NetGAN: reconstruction step Generator 5 } 3 1 Generate many synthetic } 1 4 4 1 6 ,..., 1 random walks 9 1 1 7 1 2 Count transitions in score matrix S Transform S into edge probability matrix Sample edges 1 3 1 8 4 1 1 without replacement 5 1 6 9 1 1 2 4 / 14 1 7

NetGAN: reconstruction step Generator 5 } 3 1 Generate many synthetic } 1 4 4 1 6 ,..., 1 random walks 9 1 1 7 1 2 Count transitions in score matrix S Transform S into edge probability matrix Sample edges 1 3 1 8 4 1 1 without replacement 5 1 6 9 1 1 2 4 / 14 1 7

NetGAN: reconstruction step Generator 5 } 3 1 Generate many synthetic } 1 4 4 1 6 ,..., 1 random walks 9 1 1 7 1 2 Count transitions in score matrix S Transform S into edge probability matrix Sample edges 1 3 1 8 4 1 1 without replacement 5 1 6 9 1 1 2 4 / 14 1 7

NetGAN: reconstruction step Generator 5 } 1 3 Generate many synthetic } 4 1 4 1 6 ,..., 1 random walks 9 1 7 1 2 1 Count transitions in score matrix S Transform S into edge probability matrix Sample edges 3 1 8 1 1 4 1 without replacement 1 5 6 1 9 2 1 4 / 14 1 7

Overview of simplifications Sample Sample Count Train GAN NetGAN: random walks random walks transitions in Convert with LSTM from graph from generator score matrix score matrix Sample Input into edge- output graph graph independent model 5 / 14

Overview of simplifications Sample Sample Count Train GAN NetGAN: random walks random walks transitions in Convert with LSTM from graph from generator score matrix score matrix Sample Input into edge- Solve output graph graph Solve independent optimization eigenvector model CELL: problem problem for with rank score matrix constraint 5 / 14

Overview of simplifications Sample Sample Count Train GAN NetGAN: random walks random walks transitions in Convert with LSTM from graph from generator score matrix score matrix Sample Input into edge- Solve output graph graph Solve independent optimization eigenvector model CELL: problem problem for with rank score matrix constraint 1. Replace GAN with rank-constrained optimization problem 5 / 14