To Towards Understanding the Ge Geome metry of Knowl wledge Gr Graph Em Embed eddings Chandrahas 1 , Aditya Sharma 2 , Partha Talukdar 1,2 1 Computer Science and Automation 2 Computatational and Data Sciences Indian Institute of Science, Bangalore
Kn Knowledge Graphs (KG) FC Barcelona Argentina National Football Team Lionel Messi Has Football Team 1
Kn Knowledge Graphs (KG) FC Barcelona Entities Argentina National Football Team Lionel Messi Has Football Team 2
Kn Knowledge Graphs (KG) Relations FC Barcelona Entities Argentina National Football Team Lionel Messi Has Football Team 3
Kn Knowledge Graphs (KG) • Example KGs 4
Kn Knowledge Graphs (KG) • Example KGs • Applications • Search 5
Kn Knowledge Graphs (KG) • Example KGs • Applications • Search • Question Answering 6
KG KG Em Embe beddi ddings ngs • Represents entities and relations as vectors in a vector space ℝ 𝑒 TransE 1 7 1. Translating Embeddings for Modeling Multi-relational Data, Bordes et al.
KG KG Em Embe beddi ddings ngs • Represents entities and relations as vectors in a vector space 𝒆 ... Plays For ... ... ℝ 𝑒 TransE 1 8 1. Translating Embeddings for Modeling Multi-relational Data, Bordes et al. NIPS 2013.
Ge Geometr try of Em Embe beddi ddings ngs • Arrangement of vectors in the vector space. 9
Ge Geometr try of Em Embe beddi ddings ngs • A recent work by (Mimno and Thompson, 2017) 1 presented an analysis of the geometry of word embeddings and revealed interesting results. • However, geometrical understanding of KG embeddings is very limited, despite their popularity. 10 1. The strange geometry of skip-gram with negative sampling, Mimno and Thompson, EMNLP 2017
Pr Problem • Study the geometrical behavior of KG embeddings learnt by different methods. • Study the effect of various hyper-parameters used during training on the geometry of KG embeddings. • Study the correlation between the geometry and performance of KG embeddings. 11
KG KG Embedding Methods • Learns d-dimensional vectors for entities 𝓕 and relations 𝓢 in a KG. 12
KG KG Embedding Methods • Learns d-dimensional vectors for entities 𝓕 and relations 𝓢 in a KG. + • A score function 𝛕 : 𝓕 ⨉ 𝓢 ⨉ 𝓕 → ℝ distinguishes correct triples 𝑈 − . For example, from incorrect triples 𝑈 𝛕 ( Messi, plays-for-team, Barcelona ) > 𝛕 ( Messi, plays-for-team, Liverpool ) 13
KG KG Embedding Methods • Learns d-dimensional vectors for entities 𝓕 and relations 𝓢 in a KG. + • A score function 𝛕 : 𝓕 ⨉ 𝓢 ⨉ 𝓕 → ℝ distinguishes correct triples 𝑈 − . For example, from incorrect triples 𝑈 𝛕 ( Messi, plays-for-team, Barcelona ) > 𝛕 ( Messi, plays-for-team, Liverpool ) + , 𝑈 − ) is used for training the embeddings (usually • A loss function 𝑀(𝑈 logistic loss or margin-based ranking loss). 14
KG KG Embedding Methods 15
KG KG Embedding Methods • Additive Methods • Multiplicative Methods • Neural Methods 16
KG KG Embedding Methods 17 ☉ Entry-wise product ★ Circular correlation
Ge Geometr tric ical al Metr trics ics • Average Vector Length 18
Ge Geometr tric ical al Metr trics ics • Average Vector Length • Alignment to Mean 19
Ge Geometr tric ical al Metr trics ics • Conicity 20
Ge Geometr tric ical al Metr trics ics • Conicity • Vector Spread 21
Ge Geometr try of Em Embe beddi ddings ngs High Conicity Low Conicity 22
Expe Experiments • We study the effect of following factors on the geometry of KG Embeddings • Type of method (Additive or Multiplicative) • Number of Negative Samples • Dimension of Vector Space • We also study the correlation of performance and geometry. • For experiments, we used FB15k dataset. 23
Addi Additive vs s Mul ultipl plicative (En (Entity y Vectors) s) Additive Multiplicative 24
Addi Additive vs s Mul ultipl plicative (R (Relation n Vectors) s) Additive Multiplicative 25
Addi Additive vs s Mul ultipl plicative Model Type Conicity Vector Spread Additive Low High Multiplicative High Low 26
Ef Effect of #Negative Samples (Entity Vectors) 27
Ef Effect of #Negative Samples (Entity Vectors) Additive 28
Ef Effect of #Negative Samples (Entity Vectors) Multiplicative 29
Ef Effect of #Negative Samples (Entity Vectors) Additive No change 30
Ef Effect of #Negative Samples (Entity Vectors) Additive No change Multiplicative Conicity Increases 31
Ef Effect of #Negative Samples (Entity Vectors) Additive No change 32
Ef Effect of #Negative Samples (Entity Vectors) Additive No change Multiplicative AVL decreases 33
Ef Effect of #Negative Samples Model Type Vector Type Conicity AVL Entity No Change No Change Additive Relation No Change No Change Entity Increases Decreases Multiplicative Relation Decreases No Change except HolE 34
SG SGNS S (Word2V 2Vec 1 ) ) as s Multiplicative Model • Similar observation was made by (Mimno and Thompson, 2017) 2 for SGNS based word embeddings where higher #negatives resulted in higher conicity. • Word2Vec 1 maximizes word and context vector dot product for positive word-context pairs. • This behavior is consistent with that of multiplicative models. 1. Distributed representations of words and phrases and their compositionality, Mikolov et al. NIPS 2013 35 2. The strange geometry of skip-gram with negative sampling, Mimno and Thompson, EMNLP 2017
Ef Effect of #Dimensions (Entity Vectors) Additive No change 36
Ef Effect of #Dimensions (Entity Vectors) Additive No change Multiplicative Conicity decreases 37
Ef Effect of #Dimensions (Entity Vectors) Additive No change 38
Ef Effect of #Dimensions (Entity Vectors) Additive No change Multiplicative AVL Increases 39
Ef Effect of #Dimensions Model Type Vector Type Conicity AVL Entity No Change No Change Additive Relation No Change No Change Entity Decreases Increases Multiplicative Relation Decreases Increases 40
Co Corr rrelation b/w Geome metry y and Perf rforma rmance 41
Co Corr rrelation b/w Geome metry y and Perf rforma rmance Additive 42
Co Corr rrelation b/w Geome metry y and Perf rforma rmance HolE performs bad with higher negatives 43
Co Corr rrelation b/w Geome metry y and Perf rforma rmance Negative Slope- Negative Correlation 44
Co Corr rrelation b/w Geome metry y and Perf rforma rmance Negative Slope- Negative Correlation Higher Negatives- Higher Slope Magnitude 45
Co Corr rrelation b/w Geome metry y and Perf rforma rmance 46
Co Corr rrelation b/w Geome metry y and Perf rforma rmance Additive and HolE 47
Co Corr rrelation b/w Geome metry y and Perf rforma rmance Positive Slope- Positive Correlation 48
Co Corr rrelation b/w Geome metry y and Perf rforma rmance Positive Slope- Positive Correlation Higher Negatives- Higher Slope Magnitude 49
Co Corr rrelation b/w Geome metry y and Perf rforma rmance • Additive: No correlation between geometry and performance. • Multiplicative: For fixed number of negative samples, • Conicity has negative correlation with performance • AVL has positive correlation with performance 50
Co Conclusi sion and Future Work rks • We initiated the study of geometrical behavior of KG embeddings and presented various insights. • Explore whether other entity/relation features (eg entity category) have any correlation with geometry. • Explore other geometrical metrics which have better correlation with performance and use it for learning better KG embeddings. 51
Ac Ackno knowl wledg dgements • We thank Google for the travel grant for attending ACL 2018. • We thank MHRD India, Intel, Intuit, Google and Accenture for supporting our work. • We thank the reviewers for their constructive comments. 52
Th Thank you 53
Ef Effect of #Negative Samples (Relation Vectors) Additive No change 54
Ef Effect of #Negative Samples (Relation Vectors) Additive No change Multiplicative Conicity decreases 55
Ef Effect of #Negative Samples (Relation Vectors) Additive No change 56
Ef Effect of #Negative Samples (Relation Vectors) Additive No change Multiplicative No change except HolE 57
Ef Effect of #Dimensions (Relation Vectors) Additive No change 58
Ef Effect of #Dimensions (Relation Vectors) Additive No change Multiplicative Conicity decreases 59
Ef Effect of #Dimensions (Relation Vectors) Additive No change 60
Ef Effect of #Dimensions (Relation Vectors) Additive No change Multiplicative AVL Increases 61
Recommend
More recommend
