[PPT] - A PRIMER ON GRAPH KERNELS Karsten Borgwardt Interdepartmental PowerPoint Presentation

SLIDE 1

Graph Mining and Graph Kernels

A PRIMER ON GRAPH KERNELS

Karsten Borgwardt

Interdepartmental Bioinformatics Group Max-Planck-Institutes for Biological Cybernetics and Developmental Biology, Tübingen

Summer Semester 2009, Biological Network Analysis

SLIDE 2

Graph Mining and Graph Kernels

Graph Comparison

G G′ G

s G × G →

sG, G′ G G′

Karsten Borgwardt | Biological Network Analysis | 2

G G

SLIDE 3

Graph Mining and Graph Kernels

Applications of Graph Comparison

Karsten Borgwardt | Biological Network Analysis | 3

!

SLIDE 4

Graph Mining and Graph Kernels

Graph Isomorphism

" " " "

#"$ #"%

"$ "% &''()"$ () "%'*)"$ "%

+

Karsten Borgwardt | Biological Network Analysis | 4

+ ,+

, , , ,

, ,#

"$ "%

, +

SLIDE 5

Graph Mining and Graph Kernels

Subgraph Isomorphism

+

+ + +

,+

+)''#,,'

, , , ,

(#
Karsten Borgwardt | Biological Network Analysis |

5

(#

.(, )

,

SLIDE 6

Graph Mining and Graph Kernels

Graph Edit Distances

"$ "%

/

/), # # # #

,

Karsten Borgwardt | Biological Network Analysis |

6

, ), (,

0# 0# 0# 0#

,

SLIDE 7

Graph Mining and Graph Kernels

Topological Descriptors

!#

#

# # # #

.#

Karsten Borgwardt | Biological Network Analysis | 7

0# 0# 0# 0#

#

,

SLIDE 8

Graph Mining and Graph Kernels

Polynomial Alternatives

1 1 1 1

+

" " " "

,,

'

Karsten Borgwardt | Biological Network Analysis | 8

(#

# ,

SLIDE 9

Graph Mining and Graph Kernels

What is a Kernel? (Schölkopf ,1997)

! ,2 x x′ # φ H'
! H φx, φx′'
H

kx, x′ 3 φx, φx′'

Karsten Borgwardt | Biological Network Analysis | 9

SLIDE 10

Graph Mining and Graph Kernels

What is a Graph Kernel?

Instance of R-convolution kernels by Haussler (1999)

.+# ,2

kconvolutionx, x′ 3

kpartsxd, x′

d

Karsten Borgwardt | Biological Network Analysis | 10

kconvolutionx, x′ 3

x,x∈
x′

,x′∈

kpartsxd, x′

d

" #

)

R '
,

'' 4) ) ) + '

SLIDE 11

Graph Mining and Graph Kernels

Hardness Results on Graph Kernels

(Gaertner, Flach, Wrobel, COLT 2003)

")"’3 φ") φ"5 ,'

6φ 2#) '

Karsten Borgwardt | Biological Network Analysis | 11

φ 2#)

kG, G − %kG, G′ 7 kG′, G′

3

φG − φG′, φG − φG′

3 φG − φG′ 3 8 G G′'

SLIDE 12

Graph Mining and Graph Kernels

Random Walks (Kashima et al., ICML 2003, Gaertner et al., COLT 2003)

""’

19

1 ,,+

2(

Karsten Borgwardt | Biological Network Analysis |

12

2(

""5 "×3:×)-×

""5

SLIDE 13

Graph Mining and Graph Kernels

Random Walks – Direct Product Graph

$ % ; $′ %′

X

$, $′ $, %′

G× G G′

Karsten Borgwardt | Biological Network Analysis | 13

$, $′ $, %′ %, $′ %, %′ ;, $′ ;, %′

G×

SLIDE 14

Graph Mining and Graph Kernels

Setbacks of Random Walk Kernels

0# 0# 0# 0#

., * 5<5

Karsten Borgwardt | Biological Network Analysis | 14

: ')

6%88=

#,! ')6!%88> ",?'@)60!%88A

SLIDE 15

Graph Mining and Graph Kernels

Fast Computation of Random Walk Kernels

(Vishwanathan et al., NIPS 2006)

0B 0B 0B 0B=

= = =

Karsten Borgwardt | Biological Network Analysis |

15

#-9

*,#B;

SLIDE 16

Graph Mining and Graph Kernels

Vec-Operator and Kronecker Products

: : : :+ + + +B B B B

#((% ( $##' 6()

' @ @ @ @

?

Karsten Borgwardt | Biological Network Analysis |

16

?

(?

A ⊗ B 3    A,B A,B . . . A,nB ' ' ' ' ' ' ' ' ' ' ' ' An,B An,B . . . An,mB   

SLIDE 17

Graph Mining and Graph Kernels

Sylvester Equations

-9

X 3 SXT 7 X

Karsten Borgwardt | Biological Network Analysis | 17

"# n × n S) T) X'
B # X'
#, On'
6 , # 9 '

SLIDE 18

Graph Mining and Graph Kernels

From Sylvester Equations to Random Walk Kernels

) # 9

#X 3 #SXT 7 #X

B ( +

#SXT 3 T ⊤ ⊗ S #X

Karsten Borgwardt | Biological Network Analysis | 18

,# 9 −T ⊤ ⊗ S #X 3 #X.

#

#X 3 −T ⊤ ⊗ S− #X.

B , , #X⊤

#X⊤ #X 3 #X⊤ −T ⊤ ⊗ S− #X.

SLIDE 19

Graph Mining and Graph Kernels

From Sylvester Equations to Random Walk Kernels

6

#X⊤ #X 3 #X⊤ −T ⊤ ⊗ S− #X , X 3 ⊤

Karsten Borgwardt | Biological Network Analysis | 19

X 3 ⊤ T 3 λAG⊤ S 3 AG′ , ⊤ #X 3 ⊤ −λAG ⊗ AG′− 3 ⊤ −λA×− .

SLIDE 20

Graph Mining and Graph Kernels

Further Speed-ups for Sparse Graphs

:+*

S T , ' 1 C T ⊤ ⊗ S #X X #SXT' < ( D

(+ 6

Karsten Borgwardt | Biological Network Analysis | 20

0 4( @ ' ) %88;

# Xk 3 7T ⊤ ⊗ S # Xk

2 " "

2 # X −T ⊤ ⊗ S # X 3 ' .9 T ⊤ ⊗S # Xk R '

SLIDE 21

Graph Mining and Graph Kernels

Impact on Runtime for Kernel Computation

Karsten Borgwardt | Biological Network Analysis | 21

SLIDE 22

Graph Mining and Graph Kernels

Karsten Borgwardt | Biological Network Analysis | 22

SLIDE 23

Graph Mining and Graph Kernels

Tottering (Mahe et al., ICML 2004)

1

## @ <#

Karsten Borgwardt | Biological Network Analysis | 23

A B A B

G G‘

Tottering

SLIDE 24

Graph Mining and Graph Kernels

Preventing Tottering

-( , , % ) v, . . . , vl

vi 3 vi i ∈ {$, . . . , l − %}'

G 3 V, E

4 GT VT 3 V ∪ E ET 3 {v, v, t|v ∈

Karsten Borgwardt | Biological Network Analysis | 24

∪ { | ∈ V, v, t ∈ E} ∪ {u, v, v, t|u, v, v, t ∈ E, u 3 t} * GT # G 6 GT ) , " 2 ) , " ( )

SLIDE 25

Graph Mining and Graph Kernels

Preventing Tottering

1 GT G) , ,

, %

!4 E On On #

F

Karsten Borgwardt | Biological Network Analysis | 25

F

( # # +

4

SLIDE 26

Graph Mining and Graph Kernels

Label Enrichment: Morgan Index (1965)

E ,) 6, *(, )!6(+,

2

2 4 4

Karsten Borgwardt | Biological Network Analysis | 26

Original graph

2 2 2 2 2 2 2 2 3 3

1st order Morgan Index

4 4 5 5 5 5 4 4 7 7

2nd order Morgan Index

SLIDE 27

Graph Mining and Graph Kernels

Replacing Walks by Paths

0,

, , , ,

+ +

Karsten Borgwardt | Biological Network Analysis |

27

+ ?,B;G

,,(

, 1 1 1 1

,9), 0,

SLIDE 28

Graph Mining and Graph Kernels

Shortest-Path Kernel on Graphs (B. and Kriegel, ICDM 2005)

+++ G G′ # +1
04 , G

G′ kG, G′ 3

v,v∈G
v′

,v′ ∈G′

klengthdvi, vj, dv′

k, v′ l

Karsten Borgwardt | Biological Network Analysis | 28

v,v∈G
v′

,v′ ∈G′

dvi, vj , vi vj
klength )

) kdvi, vj, dv′

k, v′ l 3 dvi, vj ∗ dv′ k, v′ l)

kdvi, vj, dv′

k, v′ l 3

$ dvi, vj 3 dv′

k, v′ l

8

SLIDE 29

Graph Mining and Graph Kernels

Link to Wiener Index (Wiener, 1947)

G 3 V, E !" WG G WG 3

v∈G
v∈G

dvi, vj, $ dvi, vj vi

Karsten Borgwardt | Biological Network Analysis | 29

dvi, vj vi vj G

SLIDE 30

Graph Mining and Graph Kernels

Link to Wiener Index

1 6 WG WG′

WG ∗ WG′ 3

v∈G
v∈G

dvi, vj

v′

∈G′

v′

∈G′

dv′

k, v′ l

3

v∈G
v∈G
v′

∈G′

v′

∈G′

dvi, vjdv′

k, v′ l

Karsten Borgwardt | Biological Network Analysis |

30

v∈G
v∈G
v′

∈G′

v′

∈G′

3

v,v∈G
v′

,v′ ∈G′

klineardvi, vj, dv′

k, v′ l

3 kshortest pathG, G′

SLIDE 31

Graph Mining and Graph Kernels

Properties of Shortest-Path Kernel

# # # #

),, .B>

–+++""‘B; –""‘B>

Karsten Borgwardt | Biological Network Analysis | 31

,,

E 0# 0# 0# 0#

B> 0()

,

SLIDE 32

Graph Mining and Graph Kernels

Optimal Assignment Kernel (Froehlich et al., ICML 2005)

G G′
{x, . . . , x|G|} , G) ''
{y, . . . , y|G′|} , G′) ''
k +# ,

Karsten Borgwardt | Biological Network Analysis | 32

π , {$, . . . , |G|, |G′|}
*

kAG, G′ 3

(π

|G|

i kxi, yπi,

|G′| ≥ |G| (π |G′|

j kxπj, yj,

) 6! %88A
# 4 :) %88H

SLIDE 33

Graph Mining and Graph Kernels

Weighted Decomposition Kernel (Menchetti et al., ICML 2005)

G 3 V, E G′ 3 V ′, E′
6 4 F ,
, " ) δ
z 3 z, ..., zD , G !

" x) κ

Karsten Borgwardt | Biological Network Analysis | 33

*

kG, G′ 3

s,z∈R−G,s′,z′∈R−G′

δs, s′

D

d

κzd, z′

d

$ # ! ') 6! %88A

-( s , z , s G

SLIDE 34

Graph Mining and Graph Kernels

Edit-Distance Kernel (Neuhaus and Bunke, 2006)

* ,

. 6 , +

,

Karsten Borgwardt | Biological Network Analysis | 34

# # # #

+

(# 0# 0# 0# 0#

*+ #

SLIDE 35

Graph Mining and Graph Kernels

Subtree Kernel (Ramon and Gaertner, 2004)

,+

,+,

#

#""‘

–##

Karsten Borgwardt | Biological Network Analysis | 35

## –.#,##

#

# # #

.+,

0# 0# 0# 0#

.(,+

SLIDE 36

Graph Mining and Graph Kernels

Cyclic Pattern Kernel (Horvath et al., KDD 2004)

9
,(,#
0)
Karsten Borgwardt | Biological Network Analysis |

36

#

# # #

6#+,

0# 0# 0# 0#

+ .,

,,

SLIDE 37

Graph Mining and Graph Kernels

Graphlet Kernel (B., Petri, et al., MLG 2007)

,E " "‘

*,

E2)?

%88I ., ., ., .,

Karsten Borgwardt | Biological Network Analysis |

37

., ., ., .,

(# ,B

, , , , 0# 0# 0# 0#

,,

SLIDE 38

Graph Mining and Graph Kernels

Graphlet Kernel

Karsten Borgwardt | Biological Network Analysis | 38

SLIDE 39

Graph Mining and Graph Kernels

Recent Trends

,?)6!%88H ,?)6!%88H ,?)6!%88H ,?)6!%88H

%0;0 , ,,, E,,'

Karsten Borgwardt | Biological Network Analysis | 39

,@?')6!%88H ,@?')6!%88H ,@?')6!%88H ,@?')6!%88H

.# 0## ,#,B;

SLIDE 40

Graph Mining and Graph Kernels

Applications: Chemoinformatics (Ralaivola et al., 2005)

", ", ", ",

0*)!!()<,

?

, +

Karsten Borgwardt | Biological Network Analysis | 40

,#(

*)
(E

#'

SLIDE 41

Graph Mining and Graph Kernels

Chemical Compound Classification (Wale et al, ICDM 2006)

(

( ( (

(

( ( ( 9 9 9 9

0 ,

(

+, 9

, +,

Karsten Borgwardt | Biological Network Analysis |

41

" # ,

(

‘‘ , # ( # 9

,

SLIDE 42

Graph Mining and Graph Kernels

Applications: Protein Function Prediction (B. et al, ISMB 2005)

! :!

.#,

Karsten Borgwardt | Biological Network Analysis | 42

SLIDE 43

Graph Mining and Graph Kernels

Future Challenges for Graph Kernel Research

0# 0# 0# 0#

!

# # # #

,

Karsten Borgwardt | Biological Network Analysis |

43

, (

6# 6# 6# 6#

), ,6

SLIDE 44

Graph Mining and Graph Kernels

References

Francis Bach: Graph kernels between point clouds. ICML 2008 Karsten M. Borgwardt, Hans-Peter Kriegel: Shortest-Path Kernels on

Graphs. ICDM 2005: 74-81

Karsten M. Borgwardt, Cheng Soon Ong, Stefan Schönauer, S. V. N.

Vishwanathan, Alexander J. Smola, Hans-Peter Kriegel: Protein function prediction via graph kernels. ISMB (Supplement of Bioinformatics) 2005: 47-56

Karsten Borgwardt | Biological Network Analysis | 44

47-56

Karsten M. Borgwardt, Tobias Petri, S. V. N. Vishwanathan, Hans-Peter

Kriegel: An Efficient Sampling Scheme For Comparison of Large Graphs. MLG 2007

Mukund Deshpande, Michihiro Kuramochi, Nikil Wale, George Karypis:

Frequent Substructure-Based Approaches for Classifying Chemical

Compounds. IEEE Trans. Knowl. Data Eng. 17(8): 1036-1050 (2005)

Holger Fröhlich, Jörg K. Wegner, Florian Sieker, Andreas Zell: Optimal

assignment kernels for attributed molecular graphs. ICML 2005: 225-232

SLIDE 45

Graph Mining and Graph Kernels

References

Thomas Gärtner, Peter A. Flach, Stefan Wrobel: On Graph Kernels:

Hardness Results and Efficient Alternatives. COLT 2003: 129-143

David Haussler. Convolution kernels on discrete structures. UCSC-CRL-

99-10,1999.

Tamás Horváth, Thomas Gärtner, Stefan Wrobel: Cyclic pattern kernels

for predictive graph mining. KDD 2004: 158-167

Hisashi Kashima, Koji Tsuda, Akihiro Inokuchi: Marginalized Kernels

Karsten Borgwardt | Biological Network Analysis | 45

Hisashi Kashima, Koji Tsuda, Akihiro Inokuchi: Marginalized Kernels

Between Labeled Graphs. ICML 2003: 321-328

Imre Risi Kondor, Karsten M. Borgwardt: The skew spectrum of graphs.

ICML 2008

Pierre Mahé, Nobuhisa Ueda, Tatsuya Akutsu, Jean-Luc Perret, Jean-

Philippe Vert: Extensions of marginalized graph kernels. ICML 2004

SLIDE 46

Graph Mining and Graph Kernels

References

Sauro Menchetti, Fabrizio Costa, Paolo Frasconi: Weighted

decomposition kernels. ICML 2005:585-592

Michel Neuhaus, Horst Bunke: A Random Walk Kernel Derived from

Graph Edit Distance. SSPR/SPR 2006: 191-199

Liva Ralaivola, Sanjay Joshua Swamidass, Hiroto Saigo, Pierre Baldi:

Graph kernels for chemical informatics. Neural Networks 18(8): 1093-

Karsten Borgwardt | Biological Network Analysis | 46

Graph kernels for chemical informatics. Neural Networks 18(8): 1093- 1110 (2005)

Jan Ramon, Thomas Gärtner: Expressivity versus Efficiency of Graph

Kernels. First International Workshop on Mining Graphs, Trees and

Sequences 2003

S.V.N. Vishwanathan, Karsten M. Borgwardt, Nicol N. Schraudolph: Fast

Computation of Graph Kernels. NIPS 2006:1449-1456

Nikil Wale, George Karypis: Comparison of Descriptor Spaces for

Chemical Compound Retrieval and Classification. ICDM 2006: 678-689