An Introduction to Graph Kernels Karsten Borgwardt and Oliver Stegle - PowerPoint PPT Presentation

An Introduction to Graph Kernels Karsten Borgwardt and Oliver Stegle Machine Learning and Computational Biology Research Group, Max Planck Institute for Biological Cybernetics and Max Planck Institute for Developmental Biology, Tübingen Karsten Borgwardt and Oliver Stegle: Computational Approaches for Analysing Complex Biological Systems, Page 1

Graph Comparison Graph Mining and Graph Kernels !"#$%&%'$ ( )*+,-. /'0-,+%1'$ 2+'34"05 !"#$% &'( )*+,-. ! +%/ ! ! 0*(1 &-$ .,+2$ (0 )*+,-. ! 3 4-$ ,*(56$1 (0 )*+,- 2(1,+*".(% ". &( 7%/ + 1+,,"%) " ! ! " ! # ! .82- &-+& " " !# ! ! # 98+%&"7$. &-$ ."1"6+*"&: ;(* /".."1"6+*"&:< (0 ! 8%/ ! ! 3 G G´ An Introduction to Graph Kernels 2

Applications of Graph Comparison Graph Mining and Graph Kernels � Function prediction of chemical compounds � Structural comparison and function prediction of protein structures � Comparison of social networks � Analysis of semantic structures in Natural Language Processing � Comparison of UML diagrams An Introduction to Graph Kernels 3

Graph Isomorphism Graph Mining and Graph Kernels Graph isomorphism � Find a mapping f of the vertices of G 1 to the vertices of G 2 such that G 1 and G 2 are identical; i.e. (x,y) is an edge of G 1 iff (f(x),f(y)) is an edge of G 2 . Then f is an isomorphism, and G 1 and G 2 are called isomorphic � No polynomial-time algorithm is known for graph isomorphism � Neither is it known to be NP-complete Subgraph isomorphism � Subgraph isomorphism asks if there is a subset of edges and vertices of G 1 that is isomorphic to a smaller graph G 2 � Subgraph isomorphism is NP-complete An Introduction to Graph Kernels 4

Subgraph Isomorphism Graph Mining and Graph Kernels NP-completeness � A decision problem C is NP-complete iff � C is in NP � C is NP-hard, i.e. every other problem in NP is reducible to it. Problems for the practitioner � Excessive runtime in worst case � Runtime may grow exponentially with the number of nodes � For larger graphs with many nodes and for large datasets of graphs, this is an enormous problem An Introduction to Graph Kernels 5

Graph Edit Distances Graph Mining and Graph Kernels Principle � Count operations that are necessary to transform G 1 into G 2 � Assign costs to different types of operations (edge/node insertion/deletion, modification of labels) Advantages � Captures partial similarities between graphs � Allows for noise in the nodes, edges and their labels � Flexible way of assigning costs to different operations Disadvantages � Contains subgraph isomorphism check as one intermediate step � Choosing cost function for different operations is difficult An Introduction to Graph Kernels 6

Topological Descriptors Graph Mining and Graph Kernels Principle � Map each graph to a feature vector � Use distances and metrics on vectors for learning on graphs Advantages � Reuses known and efficient tools for feature vectors Disadvantages � Efficiency comes at a price: feature vector transformation leads to loss of topological information (or includes subgraph isomorphism as one step) An Introduction to Graph Kernels 7

Polynomial Alternatives Graph Mining and Graph Kernels Wanted � Polynomial-time similarity measure for graphs Graph kernels � Compare substructures of graphs that are computable in polynomial time. Criteria for a good graph kernel � Expressive � Efficient to compute � Positive definite � Applicable to wide range of graphs An Introduction to Graph Kernels 8

What is a Kernel? (Schölkopf ,1997) Graph Mining and Graph Kernels ! !"# $%& &'()*$+ ! ",- ! ! ./" 0"##/,1 " /,$& 2)"$34) +#"*) " 5 ! !)"+34) $6)/4 +/0/7"4/$8 /, " "+ # " 9 ! : # " 9 ! ! : $ 5 ! !"#$"% &#'() ; <&0#3$) /,,)4 #4&-3*$ /, " "+ =)4,)7 /, /,#3$ +#"*) $ 9 !# ! ! : > # " 9 ! : # " 9 ! ! : $ 5 An Introduction to Graph Kernels 9

What is a Graph Kernel? Graph Mining and Graph Kernels Instance of R-convolution kernels by Haussler (1999) ! !"#$%&$'()*$% +,-%,'. #$/01-, 2,#$/0$.*)*$%. $3 )4$ .)-(#)(-,2 $56,#). ! ! ! ! ! !"#$"%&'("# 7 "# " ! 8 9 ! !"#$"%&'("# 7 "# " ! 8 9 ! )*+', 7 " - # " ! ! )*+', 7 " - # " ! - 8 - 8 ! . ! ! /. ! " " # ! . ! /. " " # ! :-10; +,-%,'. 1-, #$%&$'()*$% +,-%,'. $% 01*-. $3 <-10;. 7 !"# 01*-. $3 %$2,.= );$(<; );*. *. 1 #$//$% (., *% );, '*),-1)(-,8 ! > %,4 2,#$/0$.*)*$% -,'1)*$% $ -,.('). *% 1 %,4 <-10; +,-%,'? ! > <-10; +,-%,' /1+,. );, 4;$', 31/*'@ $3 +,-%,' /,);$2. 100'*#15', )$ <-10;. 7,?<? 3$- #'1..*A#1)*$%= #'(.),-*%<= 3,1)(-, .,',#)*$%= )4$".1/0', ),.).8? An Introduction to Graph Kernels 10

Hardness Results (Gaertner, Flach, Wrobel, COLT 2003) Graph Mining and Graph Kernels !"#$%&'%()*+,% !"#$%&'%()*+,%"-'.')+,"-. !"#$%&'%()*+,% !"#$%&'%()*+,% "-'.')+,"-. "-'.')+,"-. "-'.')+,"-. ! !/&%$0121 ’ 3%4 ! ! 0132 ! 0153 " 6/%*%()*+,%$/)#/78% ! 9:% ! "-%"#;/<&"=/2% $ "-%<*77/>%*%<'.+7/&/%()*+,%$/)#/78 !"#$#%&'&#( ) !"#$%&'() *(+ ,"#$-.&. )/*$0 1./(.- '2 *& -.*2& *2 0*/3 *2 !"#$#%&'&#( ) !"#$%&'() *(+ ,"#$-.&. )/*$0 1./(.- '2 *& -.*2& *2 0*/3 *2 3.,'3'() 40.&0./ &4" )/*$02 */. '2"#"/$0',5 !"##* ?- ! "- "#;/<&"=/2 ! " 0 #$ # 3 ! @ " 0 #$ # ! 3 A " 0 # ! $ # ! 3 ! " ! 0 # 3 ! ! 0 # ! 3 $ ! 0 # 3 ! ! 0 # ! 3 # 4 4 $ ! 0 # 3 ! ! 0 # ! 3 $ 4 B ": *#> '#7C ": # "- "-'.')+,"< &' # ! 8 An Introduction to Graph Kernels 11

Random Walks (Kashima et al., ICML 2003, Gaertner et al., COLT 2003) Graph Mining and Graph Kernels Principle � Count common walks in two input graphs G and G’ � Walks are sequences of nodes that allow repetitions of nodes Elegant computation � Walks of length k can be computed by looking at the k -th power of the adjacency matrix � Construct direct product graph of G and G' � Count walks in this product graph G x =(V x ,E x ) � Each walk in the product graph corresponds to one walk in G and G' An Introduction to Graph Kernels 12

Random Walks – Direct Product Graph Graph Mining and Graph Kernels ! " ! " X ! ! " ! " # " " " " ! ! ! ! ! ! ! ! ! ! " ! ! ! " ! " ! ! ! " ! " ! # ! ! ! # ! " ! An Introduction to Graph Kernels 13

Setbacks of Random Walk Kernels Graph Mining and Graph Kernels Disadvantages � Runtime problems � Tottering � 'Halting' Potential solutions � Fast computation of random walk graph kernels (Vishwanathan et al., NIPS 2006) � Preventing tottering and label enrichment (Mahe et al., ICML 2004) � Graph kernels based on shortest paths (B. and Kriegel, ICDM 2005) An Introduction to Graph Kernels 14

Fast Random Walk Kernels Graph Mining and Graph Kernels (Vishwanathan et al., NIPS 2006) Direct computation: O(n 6 ) Solution � Cast computation of random walk kernel as Sylvester Equation � These can be solved in O(n 3 ) An Introduction to Graph Kernels 15

Vec-Operator and Kronecker Products Graph Mining and Graph Kernels Vec-Operator � vec flattens an n x n matrix A into an n 2 x 1 vector vec(A). � It stacks the columns of the matrix on top of each other, from left to right. Kronecker Product � Product of two matrices A and B � Each element of A is multiplied with the full matrix B: ! $ ! ! ! ! " ! ! ! " " # # # ! ! !" " " % 9 9 9 9 9 9 9 9 ! ! " FG # & 9 9 9 9 ! "! ! " ! "! " " # # # ! "!# " An Introduction to Graph Kernels 16

Sylvester Equations Graph Mining and Graph Kernels ! !"#$%&'() '* %+, *'-. ! / "!# 0 ! ! ! 1&2,( %+-,, $ " $ .$%-&3,) " 4 # 4 $(5 ! ! 6 ! 7(, 8$(%) %' )'92, *'- ! 6 ! :'92$;9, &( % < $ " =6 ! >% &) ?'))&;9, %' %#-( :@92,)%,- ,"#$%&'() &(%' A-$?+ B,-(,9)6 An Introduction to Graph Kernels 17

From Sylvester Equations to Random Walk Kernels Graph Mining and Graph Kernels ! !"#$%& %'( )*+,($%(# (-./%"01 "$ #(2#"%%(1 /$ ,(34 ! 5 6 ,(34 "!# 5 7 ,(34 ! ! 5 ! 81( %'(1 (9:+0"%$ %'( 2(++;<1021 =/3% ,(34 "!# 5 6 4 # ! " " 5 ,(34 ! 5 %0 #(2#"%( %'( />0,( -.($%"01 /$ 4 ! # # ! " " 5 ,(34 ! 5 6 ,(34 ! ! 5 $ ! ?02 01( '/$ %0 $0+,( ,(34 ! 5 6 4 ! # # ! " " 5 " " ,(34 ! ! 5 $ ! 81( @.+%":+"($ >0%' $"A($ >* ,(34 ! ! 5 ! ,(34 ! ! 5 ! ,(34 ! 5 6 ,(34 ! ! 5 ! 4 ! # # ! " " 5 " " ,(34 ! ! 5 $ An Introduction to Graph Kernels 18

From Sylvester Equations to Random Walk Kernels Graph Mining and Graph Kernels ! !" #$%& ! ! ' ! #$%& ! ' ( #$%& ! ! ' ! & ! " " ! # # ' " " #$%& ! ! ' )"$ *+,*-.-+-$* ! ! ( " " ! ! ! ( " " ! " ( $% & & ' ! # ( % & & # ' /"0 ),-/." " ! #$%& ! ' ( " ! & ! " $% & & ' # % & & # '' " " " ( " ! & ! " $% $ ' " " " ' An Introduction to Graph Kernels 19