http://cs224w.stanford.edu 1. Introduction to Knowledge Graphs 2. - PowerPoint PPT Presentation

CS224W: Machine Learning with Graphs Jure Leskovec, Hongyu Ren, Stanford University http://cs224w.stanford.edu

1. Introduction to Knowledge Graphs 2. Knowledge Graph completion 3. Path Queries 4. Conjunctive Queries 5. Query2Box: Reasoning with Box Embeddings 11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 2

¡ Knowledge in graph form § Capture entities, types, and relationships ¡ Nodes are entities ¡ Nodes are labeled with their types ¡ Edges between two nodes capture relationships between entities 11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 3

¡ Node types : paper, title, author, conference, year ¡ Relation types : pubWhere, pubYear, hasTitle, hasAuthor, cite 11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 4

¡ Node types : account, song, post, food, channel ¡ Relation types : friend, like, cook, watch, listen 11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 5

paintedBy 11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 6

¡ Google Knowledge Graph ¡ Amazon Product Graph ¡ Facebook Graph API ¡ IBM Watson ¡ Microsoft Satori ¡ Project Hanover/Literome ¡ LinkedIn Knowledge Graph ¡ Yandex Object Answer 11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 7

¡ Serving information 11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 8

¡ Question answering and conversation agents 11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 9

1. Introduction to Knowledge Graphs 2. Knowledge Graph completion 3. Path Queries 4. Conjunctive Queries 5. Query2Box: Reasoning with Box Embeddings 11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 10

¡ Publicly available KGs: § FreeBase, Wikidata, Dbpedia, YAGO, NELL, etc. ¡ Common characteristics: § Massive : millions of nodes and edges § Incomplete : many true edges are missing Given a massive KG, enumerating all the Can we predict plausible possible facts is BUT missing links? intractable! 11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 11

¡ Freebase § ~50 million entities 93.8% of persons from Freebase have no place of birth and 78.5% § ~38K relation types have no nationality! § ~3 billion facts/triples ¡ FB15k/FB15k-237 § A complete subset of Freebase, used by researchers to learn KG models [1] Paulheim, Heiko. "Knowledge graph refinement: A survey of approaches and evaluation methods." Semantic web 8.3 (2017): 489-508. [2] Min, Bonan, et al. "Distant supervision for relation extraction with an incomplete knowledge base." Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . 2013. 11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 12

¡ Given an enormous KG, can we complete the KG / predict missing relations? § links + type missing relation: genre 11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 13

¡ Edges in KG are represented as triples (ℎ, 𝑠, 𝑢) § head (ℎ) has relation 𝑠 with tail (𝑢) . ¡ Key Idea: § Model entities and relations in the embedding/vector space ℝ ( . § Given a true triple (ℎ, 𝑠, 𝑢) , the goal is that the embedding of (ℎ, 𝑠) should be close to the embedding of 𝑢 . § How to embed ℎ, 𝑠 ? § How to define closeness? 11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 14

¡ Symmetric Relations: 𝑠 ℎ, 𝑢 ⇒ 𝑠 𝑢, ℎ ∀ℎ, 𝑢 § Example : Family, Roommate ¡ Composition Relations: 𝑠 + 𝑦, 𝑧 ∧ 𝑠 / 𝑧, 𝑨 ⇒ 𝑠 1 𝑦, 𝑨 ∀𝑦, 𝑧, 𝑨 § Example : My mother’s husband is my father. ¡ 1-to-N, N-to-1 relations: 𝑠 ℎ, 𝑢 + , 𝑠 ℎ, 𝑢 / , … , 𝑠(ℎ, 𝑢 3 ) are all True. § Example : 𝑠 is “StudentsOf” 11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 15

¡ Translation Intuition : NOTATION: embedding For a triple (ℎ, 𝑠, 𝑢) , 𝐢, 𝐬, 𝐮 ∈ ℝ ( , vectors will appear in 𝐢 + 𝐬 = 𝐮 boldface Score function: 𝑔 ; ℎ, 𝑢 = ||ℎ + 𝑠 − 𝑢|| Nationality 𝐬 Obama 𝐢 𝐮 American Bordes, Antoine, et al. "Translating embeddings for modeling multi-relational data." Advances in neural information processing systems . 2013. 11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 16

¡ Translation Intuition: for a triple (ℎ, 𝑠, 𝑢) , 𝐢 + 𝐬 = 𝐮 Max margin loss: ; (ℎ, 𝑢 F ) G ℒ = ? 𝛿 + 𝑔 ; (ℎ, 𝑢) − 𝑔 (@,;,A)∈B,(@,;,A C )∉B Valid triple Corrupted triple where 𝛿 is the margin, i.e., the smallest distance tolerated by the model between a valid triple and a corrupted one. NOTE : check lecture 7 for a more in-depth discussion of TransE! 11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 17

¡ Who has won the Turing award? Answers! Turing Award Pearl Win 𝐫 Hinton Bengio Canada Trudeau Bieber ¡ Who is a Canadian citizen? Turing Award Pearl Answers! Hinton Bengio 𝐫 Canada Citizen Trudeau Bieber 11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 18

¡ Composition Relations: 𝑠 + 𝑦, 𝑧 ∧ 𝑠 / 𝑧, 𝑨 ⇒ 𝑠 1 𝑦, 𝑨 ∀𝑦, 𝑧, 𝑨 ¡ Example : My mother’s husband is my father. ¡ In TransE: 𝑠 1 = 𝑠 + + 𝑠 / ü 𝐳 𝐬 / 𝐬 + 𝐴 𝐲 𝐬 1 11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 19

¡ Symmetric Relations: 𝑠 ℎ, 𝑢 ⇒ 𝑠 𝑢, ℎ ∀ℎ, 𝑢 ¡ Example : Family, Roommate ¡ In TransE: 𝑠 = 0, ℎ = 𝑢 û If we want TransE to handle symmetric relations 𝑠 , for all ℎ, 𝑢 that satisfy 𝑠(ℎ, 𝑢) , 𝑠(𝑢, ℎ) is also True, which means ‖ ℎ + 𝑠 − 𝐬 ‖ 𝑢 = 0 and 𝑢 + 𝑠 − ℎ = 0 . Then 𝑠 = 0 and ℎ = 𝑢 , however ℎ and 𝑢 are two different entities and should be mapped to different 𝐢 𝐮 locations. 11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 20

¡ 1-to-N, N-to-1, N-to-N relations. ¡ Example : (ℎ, 𝑠, 𝑢 + ) and (ℎ, 𝑠, 𝑢 / ) both exist in the knowledge graph, e.g., 𝑠 is “StudentsOf” With TransE, 𝑢 + and 𝑢 / will map to the same vector, although they are different entities. 𝐮 + ¡ 𝐮 + = 𝐢 + 𝐬 = 𝐮 / 𝐬 𝐮 / ¡ 𝐮 + ≠ 𝐮 / contradictory! 𝐬 𝐢 11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 21

¡ TransR: model entities as vectors in the entity space ℝ ( and model each relation as vector 𝒔 in relation space ℝ P with 𝐍 ; ∈ ℝ P×( as the projection matrix. ¡ ℎ S = 𝑁 ; ℎ, 𝑢 S = 𝑁 ; 𝑢 ¡ 𝑔 ; ℎ, 𝑢 = ||ℎ S + 𝑠 − 𝑢 S || 𝐮 𝐢 S 𝐮 S 𝐬 𝐢 Lin, Yankai, et al. "Learning entity and relation embeddings for knowledge graph completion." AAAI . 2015. 11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 22

¡ Symmetric Relations: 𝑠 ℎ, 𝑢 ⇒ 𝑠 𝑢, ℎ ∀ℎ, 𝑢 ¡ Example : Family, Roommate 𝑠 = 0, ℎ S = 𝑁 ; ℎ = 𝑁 ; 𝑢 = 𝑢 S ü 𝐮 For TransR, we can map ℎ and 𝑢 to the same location on the space of relation 𝑠 . 𝑵 ; 𝐮 S , ℎ S 𝐢 11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 23

¡ 1-to-N, N-to-1, N-to-N relations ¡ Example : If (ℎ, 𝑠, 𝑢 + ) and (ℎ, 𝑠, 𝑢 / ) exist in the knowledge graph. We can learn 𝑁 ; so that 𝑢 S = 𝑁 ; 𝑢 + = 𝑁 ; 𝑢 / , note that 𝑢 + does not need to be equal to 𝑢 / ! 𝐮 + 𝐮 / 𝐢 S 𝐮 S 𝐬 𝐢 11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 24

¡ Composition Relations: 𝑠 + 𝑦, 𝑧 ∧ 𝑠 / 𝑧, 𝑨 ⇒ 𝑠 1 𝑦, 𝑨 ∀𝑦, 𝑧, 𝑨 ¡ Example : My mother’s husband is my father. Each relation has different space. It is not naturally compositional for multiple relations! û 11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 25

𝒈 𝒔 (𝒊, 𝒖) Embedding Entity Relation ℎ, 𝑢 ∈ ℝ ( 𝑠 ∈ ℝ ( ||ℎ + 𝑠 − 𝑢|| TransE ℎ, 𝑢 ∈ ℝ ( 𝑠 ∈ ℝ P , 𝑁 ; ∈ ℝ P×( ||𝑁 ; ℎ + 𝑠 − 𝑁 ; 𝑢|| TransR Embedding Symmetry Composition One-to-many û ü û TransE ü û ü TransR 11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 26

1. Introduction to Knowledge Graphs 2. Knowledge Graph completion 3. Path Queries 4. Conjunctive Queries 5. Query2Box: Reasoning with Box Embeddings 11/21/19 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 27