mixhop higher order graph convolutional architectures via
play

MixHop: Higher-Order Graph Convolutional Architectures via - PowerPoint PPT Presentation

Sep 06, 2023 •127 likes •608 views

1 2 MixHop: Higher-Order Graph Convolutional Architectures via Sparsified Neighborhood Mixing Sami Abu-El-Haija 1 , Bryan Perozzi 2 , Amol Kapoor 2 , Nazanin Alipourfard 1 , Kristina Lerman 1 , Hrayr Harutyunyan 1 , Greg Ver Steeg 1 , Aram

  1. 1 2 MixHop: Higher-Order Graph Convolutional Architectures via Sparsified Neighborhood Mixing Sami Abu-El-Haija 1 , Bryan Perozzi 2 , Amol Kapoor 2 , Nazanin Alipourfard 1 , Kristina Lerman 1 , Hrayr Harutyunyan 1 , Greg Ver Steeg 1 , Aram Galstyan 1 Code: http://github.com/samihaija/mixhop Slides: http://sami.haija.org/icml19 Abu-El-Haija et al, MixHop, ICML’19 Poster #88 Poster #88

  2. Agenda ● Review Graph Convolutional Networks (GCN) ○ Application Semi-Supervised Node Classification (SSNC) ○ Shortcoming of GCN ● MixHop: Higher-Order GCN ○ Sparsification ● MixHop Results on SSNC Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  3. Agenda ● Review Graph Convolutional Networks (GCN) ○ Application Semi-Supervised Node Classification (SSNC) ○ Shortcoming of GCN ● MixHop: Higher-Order GCN ○ Sparsification ● MixHop Results on SSNC Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  4. Graph Convolutional Network (GCN) [1] [1] Kipf & Welling, ICLR 2017 Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  5. Graph Convolutional Network (GCN) [1] x 4 x 5 x 2 x 3 x 6 x 1 [1] Kipf & Welling, ICLR 2017 Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  6. Graph Convolutional Network (GCN) [1] x 4 x 5 x 2 x 3 x 6 x 1 Input Features [1] Kipf & Welling, ICLR 2017 Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  7. Graph Convolutional Network (GCN) [1] x 4 x 5 x 2 x 3 x 6 GC Layer 1 x 1 Input Features [1] Kipf & Welling, ICLR 2017 Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  8. Graph Convolutional Network (GCN) [1] (1) h 4 x 4 x 5 (1) h 5 x 2 x 3 (1) h 3 x 6 (1) h 6 (1) h 2 GC Layer 1 x 1 (1) h 1 Input Features Latent Features [1] Kipf & Welling, ICLR 2017 Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  9. Graph Convolutional Network (GCN) [1] (1) h 4 x 4 x 5 (1) h 5 x 2 x 3 (1) h 3 x 6 (1) h 6 … (1) h 2 GC Layer 1 x 1 (1) h 1 Input Features Latent Features [1] Kipf & Welling, ICLR 2017 Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  10. Graph Convolutional Network (GCN) [1] (1) (L) h 4 h 4 x 4 x 5 (1) (L) h 5 h 5 x 2 x 3 (1) (L) h 3 h 3 x 6 (1) (L) h 6 h 6 … (1) (L) h 2 h 2 GC Layer 1 GC Layer L x 1 (1) (L) h 1 h 1 Input Features Latent Features Output Features [1] Kipf & Welling, ICLR 2017 Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  11. Graph Convolutional Network (GCN) [1] Train on semi-supervised node classification: (L) h 4 y 4 (L) h 5 ● measure Loss on labeled nodes ( y 4 , y 2 ) (L) h 3 (L) h 6 (L) h 2 y 2 (L) h 1 Output Features [1] Kipf & Welling, ICLR 2017 Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  12. Graph Convolutional Network (GCN) [1] Train on semi-supervised node classification: Loss (L) h 4 y 4 (L) h 5 ● measure Loss on labeled nodes ( y 4 , y 2 ) (L) h 3 ● Backprop to learn GC layers. (L) h 6 Loss (L) h 2 y 2 (L) h 1 Output Features [1] Kipf & Welling, ICLR 2017 Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  13. Graph Convolutional Network (GCN) [1] Loss (1) (L) h 4 h 4 x 4 y 4 x 5 (1) (L) h 5 h 5 x 2 x 3 (1) (L) h 3 h 3 x 6 (1) (L) h 6 h 6 … Loss (1) (L) h 2 h 2 y 2 GC Layer 1 GC Layer L x 1 (1) (L) h 1 h 1 update update Input Features Latent Features Output Features SGD [1] Kipf & Welling, ICLR 2017 Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  14. Graph Convolutional Network (GCN) [1] (1) (L) h 4 h 4 x 4 y 4 x 5 (1) (L) h 5 h 5 x 2 x 3 (1) (L) h 3 h 3 x 6 (1) (L) h 6 h 6 … (1) (L) h 2 h 2 y 2 GC Layer 1 GC Layer L x 1 (1) (L) h 1 h 1 ? [1] Kipf & Welling, ICLR 2017 Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  15. Graph Convolutional Network (GCN) [1] (1) h 4 x 4 x 5 (1) h 5 x 2 x 3 (1) h 3 x 6 (1) h 6 (1) h 2 GC Layer 1 x 1 (1) h 1 [1] Kipf & Welling, ICLR 2017 Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  16. Graph Convolutional Network (GCN) [1] (1) h 4 x 4 x 5 (1) h 5 x 3 x 2 (1) h 3 x 6 (1) h 6 Avg fc (1) h 2 GC Layer 1 x 1 (1) h 1 [1] Kipf & Welling, ICLR 2017 Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  17. Graph Convolutional Network (GCN) [1] Tensor Graph (1) h 4 x 4 x 5 (1) h 5 x 2 x 3 (1) h 3 x 6 (1) h 6 Avg fc (1) h 2 GC Layer 1 x 1 (1) h 1 [1] Kipf & Welling, ICLR 2017 Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  18. Graph Convolutional Network (GCN) [1] Tensor Graph (1) h 4 x 4 x 5 (1) h 5 x 2 x 3 (1) h 3 x 6 (1) h 6 Avg fc (1) h 2 GC Layer 1 x 1 (1) h 1 [1] Kipf & Welling, ICLR 2017 Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  19. Graph Convolutional Network (GCN) [1] Tensor Graph (1) h 4 x 4 x 5 (1) h 5 x 2 x 3 (1) h 3 x 6 (1) h 6 Avg fc (1) h 2 GC Layer 1 x 1 (1) h 1 [1] Kipf & Welling, ICLR 2017 Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  20. Graph Convolutional Network (GCN) [1] Tensor Graph (1) h 4 x 4 x 5 (1) h 5 x 2 x 3 (1) h 3 x 6 (1) h 6 Avg fc (1) h 2 GC Layer 1 x 1 (1) h 1 [1] Kipf & Welling, ICLR 2017 Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  21. Graph Convolutional Network (GCN) [1] Tensor Graph (1) h 4 x 4 x 5 (1) h 5 x 2 x 3 (1) h 3 x 6 (1) h 6 Avg fc (1) h 2 GC Layer 1 x 1 (1) h 1 [1] Kipf & Welling, ICLR 2017 Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  22. Shortcoming of Vanilla GCN Vanilla GC Layer [1] Kipf & Welling, ICLR 2017 Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  23. Shortcoming of Vanilla GCN Vanilla GC Layer 😁 fc is shared ⇒ inductive [1] Kipf & Welling, ICLR 2017 Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  24. Shortcoming of Vanilla GCN Vanilla GC Layer 😁 fc is shared ⇒ inductive 😣 Appendix Experiments of [1] shows no gains beyond 2 layers [1] Kipf & Welling, ICLR 2017 Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  25. Shortcoming of Vanilla GCN Vanilla GC Layer 😁 fc is shared ⇒ inductive 😣 Appendix Experiments of [1] shows no gains beyond 2 layers 😣 cannot mix neighbors from various distances in arbitrary linear combinations [1] Kipf & Welling, ICLR 2017 Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  26. Shortcoming of Vanilla GCN Vanilla GC Layer 😁 fc is shared ⇒ inductive 😣 Appendix Experiments of [1] shows no gains beyond 2 layers 😣 cannot mix neighbors from various distances in arbitrary linear combinations e.g. cannot learn Gabor Filters ! [1] Kipf & Welling, ICLR 2017 Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  27. Shortcoming of Vanilla GCN Vanilla GC Layer 😁 fc is shared ⇒ inductive 😣 Appendix Experiments of [1] shows no gains beyond 2 layers 😣 cannot mix neighbors from various distances in arbitrary linear combinations e.g. cannot learn Gabor Filters ! ? [1] Kipf & Welling, ICLR 2017 Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  28. Detour: Review Gabor Filters Neuroscientists discover their importance in the primate visual cortex [2, 3]: [2] Daugman, Vision Research,1980 [3] Daugman, Journal of the Optical Society of America, 1985 [4] Honglak Lee et al, ICML, 2009 [5] Alex Krizhevsky et al, NeurIPS 2012 Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  29. Detour: Review Gabor Filters Neuroscientists discover their importance in the primate visual cortex [2, 3]: Further, they are automatically recovered by training CNNs on images [4, 5] [2] Daugman, Vision Research,1980 [3] Daugman, Journal of the Optical Society of America, 1985 [4] Honglak Lee et al, ICML, 2009 [5] Alex Krizhevsky et al, NeurIPS 2012 Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  30. Main Motivation Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  31. Main Motivation Extend the class of representations realizable by GCNs e.g. to learn Gabor Filters Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  32. Agenda ● Review Graph Convolutional Networks (GCN) ○ Application Semi-Supervised Node Classification (SSNC) ○ Shortcoming of GCN ● MixHop: Higher-Order GCN ○ Sparsification ● MixHop Results on SSNC Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  33. Our Model: MixHop MixHop GC Layer Vanilla GC Layer Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  34. Our Model: MixHop MixHop GC Layer Vanilla GC Layer Couple of code lines implements concatenation Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  35. Our Model: MixHop MixHop GC Layer Vanilla GC Layer Couple of code lines implements concatenation Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  36. Our Model: MixHop MixHop GC Layer Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  37. Our Model: MixHop 😁 Inductive MixHop GC Layer Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  38. Our Model: MixHop 😁 Inductive MixHop GC Layer 😁 Can incorporate distant nodes Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  39. Our Model: MixHop 😁 Inductive MixHop GC Layer 😁 Can incorporate distant nodes 😁 Can mix neighbors across distances in arbitrary linear combinations Abu-El-Haija et al, MixHop, ICML’19 Poster #88

  40. Our Model: MixHop 😁 Inductive MixHop GC Layer 😁 Can incorporate distant nodes 😁 Can mix neighbors across distances in arbitrary linear combinations i.e. can learn Gabor Filters ! Abu-El-Haija et al, MixHop, ICML’19 Poster #88

Recommend

York University www.cs.york.ac.uk/~ndm First order vs Higher order Higher order:

York University www.cs.york.ac.uk/~ndm First order vs Higher order Higher order:

First Order Haskell Neil Mitchell York University www.cs.york.ac.uk/~ndm First order vs Higher order Higher order: functions are values Can be passed around Stored in data structure Harder for reasoning and analysis More

483 views • 15 slides
EMNLP 2017 Copenhagen Contributions } Syntactic Graph Convolutional Networks }

EMNLP 2017 Copenhagen Contributions } Syntactic Graph Convolutional Networks }

Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling Diego Marcheggiani and Ivan Titov University of Amsterdam University of Edinburgh EMNLP 2017 Copenhagen Contributions } Syntactic Graph Convolutional

865 views • 55 slides
Kipf, T., Welling, M.: Semi-Supervised Classification with Graph Convolutional Networks Radim

Kipf, T., Welling, M.: Semi-Supervised Classification with Graph Convolutional Networks Radim

Kipf, T., Welling, M.: Semi-Supervised Classification with Graph Convolutional Networks Radim petlk Czech Technical University in Prague 2 Overview - Kipf and Welling - use first order approximation in Fourier-domain to obtain an efficient

664 views • 27 slides
Why higher-order logic? Higher-Order logic Expressive Mathematics Verification

Why higher-order logic? Higher-Order logic Expressive Mathematics Verification

Extending SMT solvers to higher-order logic Haniel Barbosa Andrew Reynolds Daniel El Ouraoui Clark Barrett Cesare Tinelli SMT 2019 20190707, Lisbon, PT To be published in the proceedings of CADE 2019 Why higher-order logic?

569 views • 25 slides
On the Experimental transferability of Spectral Graph Convolutional Networks Masters project

On the Experimental transferability of Spectral Graph Convolutional Networks Masters project

On the Experimental transferability of Spectral Graph Convolutional Networks Masters project presentation 6/ 7/ 2020 Axel Nilsson Outline 1. Introduction - Spectral graph convolutional networks - ChebNet 2. Benchmarking -

317 views • 18 slides
Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation Jiaxuan You*,

Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation Jiaxuan You*,

Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation Jiaxuan You*, Bowen Liu*, Rex Ying, Vijay Pande, Jure Leskovec Stanford University 1 Motivation Question: Can we learn a model that can generate valid

502 views • 9 slides
Outline Convolutional Neural Network Architectures for Matching Natural Language Sentences.

Outline Convolutional Neural Network Architectures for Matching Natural Language Sentences.

Outline Hu, NIPS14 Irsoy, NIPS14 Outline Convolutional Neural Network Architectures for Matching Natural Language Sentences. NIPS14 Convolutional Sentence Model Convolutional Matching Models Experiments Deep Recursive Neural

779 views • 35 slides
Higher order complexity Hugo Fre Mathieu Hoyrup CCA 2013 Hugo Fre Higher order

Higher order complexity Hugo Fre Mathieu Hoyrup CCA 2013 Hugo Fre Higher order

Background & motivation Higher order strategies Higher order Turing machines Computable analysis Conclusion Higher order complexity Hugo Fre Mathieu Hoyrup CCA 2013 Hugo Fre Higher order complexity 1/12 Background &

383 views • 15 slides
Higher-order Interpretations for Higher-order Complexity Emmanuel Hainry & Romain Pchoux

Higher-order Interpretations for Higher-order Complexity Emmanuel Hainry & Romain Pchoux

Higher-order Interpretations for Higher-order Complexity Emmanuel Hainry & Romain Pchoux DICE-FOPARA, 22-23 april 2017 CNRS, INRIA, Universit de Lorraine & LORIA, Nancy, France 1 Introduction First-order computability and

801 views • 46 slides
Higher-order Interpretations for Higher-order Complexity Emmanuel Hainry & Romain Pchoux

Higher-order Interpretations for Higher-order Complexity Emmanuel Hainry & Romain Pchoux

Higher-order Interpretations for Higher-order Complexity Emmanuel Hainry & Romain Pchoux LPAR, 11 may 2017 CNRS, INRIA, Universit de Lorraine & LORIA, Nancy, France 1 Introduction First-order computability and complexity

665 views • 39 slides
Convolutional Kernel Networks for Graph-Structured Data Dexiong Chen 1 Laurent Jacob 2 Julien

Convolutional Kernel Networks for Graph-Structured Data Dexiong Chen 1 Laurent Jacob 2 Julien

Convolutional Kernel Networks for Graph-Structured Data Dexiong Chen 1 Laurent Jacob 2 Julien Mairal 1 1 Inria Grenoble 2 CNRS/LBBE Lyon ICML 2020 Dexiong Chen Graph Convolutional Kernel Networks 1 / 15 Graph-structured data are ubiquitous (a)

865 views • 28 slides
Graph Convolutional Networks

Graph Convolutional Networks

Graph_Convolutional_Networks slides 12/11/19, 8(56 AM Thanks for joining me for a presentation on ... Graph Convolutional Networks http://127.0.0.1(8000/Graph_Convolutional_Networks.slides.html?print-pdf#/ Page 1 of 30

676 views • 30 slides
Simplifying Graph Convolutional Networks Amauri Holanda Felix Wu* Tianyi Zhang* Christopher

Simplifying Graph Convolutional Networks Amauri Holanda Felix Wu* Tianyi Zhang* Christopher

Cornell University Simplifying Graph Convolutional Networks Amauri Holanda Felix Wu* Tianyi Zhang* Christopher Fifty Tao Yu Kilian Q. Weinberger de Souza Jnior* *: Equal contribution Cornell University Graph Convolutional Networks

847 views • 21 slides
Outline Higher Order Statistics First, second and higher-order statistics Matthias Hennig

Outline Higher Order Statistics First, second and higher-order statistics Matthias Hennig

Outline Higher Order Statistics First, second and higher-order statistics Matthias Hennig Generative models, recognition models Neural Information Processing Sparse Coding School of Informatics, University of Edinburgh Independent Components

358 views • 9 slides
Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks

Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks

Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks Wei-Lin Chiang 1 , Xuanqing Liu 2 , Si Si 3 , Yang Li 3 , Samy Bengio 3 , Cho-Jui Hsieh 23 1 National Taiwan University, 2 UCLA, 3 Google Research Graph

616 views • 23 slides
Some Thoughts and New Designs of Recurrent and Convolutional Architectures Fuxin Li AUGUST 1 ST

Some Thoughts and New Designs of Recurrent and Convolutional Architectures Fuxin Li AUGUST 1 ST

Some Thoughts and New Designs of Recurrent and Convolutional Architectures Fuxin Li AUGUST 1 ST , 2018 Todays Talk Multi-Target Tracking with bilinear LSTM Novel LSTM model coming from studies on tracking Understanding more about

536 views • 53 slides
CS4501: Introduction to Computer Vision Deeper Convolutional Neural Network Architectures Last

CS4501: Introduction to Computer Vision Deeper Convolutional Neural Network Architectures Last

CS4501: Introduction to Computer Vision Deeper Convolutional Neural Network Architectures Last Class Neural Networks multilayer perceptron model (MLP) Backpropagation Convolutional Neural Networks Todays Class More on

772 views • 41 slides
Graph Convolutional Network Heting Gao University of Illinois at Urbana-Champaign

Graph Convolutional Network Heting Gao University of Illinois at Urbana-Champaign

Graph Convolutional Network Heting Gao University of Illinois at Urbana-Champaign hgao17@illinois.edu November 10, 2018 Heting Gao (UIUC) Short title November 10, 2018 1 / 27 Overview Graph Convolution 1 Preliminary Graph Fourier

725 views • 27 slides
When Does Self-Supervision Help Graph Convolutional Networks? Yuning You * , Tianlong Chen * ,

When Does Self-Supervision Help Graph Convolutional Networks? Yuning You * , Tianlong Chen * ,

When Does Self-Supervision Help Graph Convolutional Networks? Yuning You * , Tianlong Chen * , Zhangyang Wang, Yang Shen Texas A&M University * Equal Contribution Department of Electrical and Computer Engineering 1 This work was presented

462 views • 10 slides
Outline Higher order is commonly used on convergence and on derivatives in opti- Trust Region with

Outline Higher order is commonly used on convergence and on derivatives in opti- Trust Region with

Workshop AD Higher Order Workshop AD Higher Order Outline Higher order is commonly used on convergence and on derivatives in opti- Trust Region with a Cubic Model mization. First order methods are gradient based and have

468 views • 7 slides
On Higher-Order Program Verification and Two Notions of Higher-Order Model Checking Naoki

On Higher-Order Program Verification and Two Notions of Higher-Order Model Checking Naoki

On Higher-Order Program Verification and Two Notions of Higher-Order Model Checking Naoki Kobayashi University of Tokyo Summaries of papers from POPL09, POPL17 (joint work with Etienne Lozes, Florian Bruse), and more recent work (joint work

963 views • 65 slides
v1 v2 vn my-map : (define (my-map f xs) F F F (F v2) (F vn) (F v1) (if (null? xs) null

v1 v2 vn my-map : (define (my-map f xs) F F F (F v2) (F vn) (F v1) (if (null? xs) null

Higher-order List Func2ons Higher-Order List Functions A function is higher-order if it takes another in Racket function as an input and/or returns another function as a result. E.g. app-3-5 , make-linear-function , flip2 from the previous

519 views • 6 slides
v1 v2 vn (if (null? xs) null F F F (cons ( f (first xs)) (F vn) (F v2) (F v1) (my-map f

v1 v2 vn (if (null? xs) null F F F (cons ( f (first xs)) (F vn) (F v2) (F v1) (my-map f

Higher-order List Func2ons Higher-Order List Functions A function is higher-order if it takes another in Racket function as an input and/or returns another function as a result. E.g. app-3-5 , make-linear-function , flip2 . We will now study

300 views • 6 slides
More JavaScript! Higher-Order Functions, Callbacks, and Array Methods Higher-Order Functions

More JavaScript! Higher-Order Functions, Callbacks, and Array Methods Higher-Order Functions

More JavaScript! Higher-Order Functions, Callbacks, and Array Methods Higher-Order Functions A Higher-Order Function is any function that operates on any other function, either by taking them in as arguments or returning them. In

308 views • 9 slides

More recommend