fofe based deep neural networks for entity discovery and
play

FOFE-based Deep Neural Networks for Entity Discovery and Linking - PowerPoint PPT Presentation

Jan 26, 2023 •188 likes •379 views

Overview What is FOFE? Entity Discovery methods FOFE-based Entity Linking Results and Conclusion FOFE-based Deep Neural Networks for Entity Discovery and Linking Nargiza Nosirova Mingbin Xu , Nargiza Nosirova , Kelvin Jiang , Feng Wei and Hui

  1. Overview What is FOFE? Entity Discovery methods FOFE-based Entity Linking Results and Conclusion FOFE-based Deep Neural Networks for Entity Discovery and Linking Nargiza Nosirova Mingbin Xu , Nargiza Nosirova , Kelvin Jiang , Feng Wei and Hui Jiang Lassonade School of Engineering, York University, Canada 2017 TAC Workshop Nargiza Nosirova FOFE-based Deep NNs for Entity Discovery & Linking

  2. Overview What is FOFE? Entity Discovery methods FOFE-based Entity Linking Results and Conclusion FOFE-based Deep Neural Networks for Entity Discovery and Linking Overview What is FOFE? FOFE-based model for Entity Discovery Ensemble modeling for Entity Discovery Multi-task model for Entity Discovery FOFE-based model for Entity Linking Experiments Conclusion Nargiza Nosirova FOFE-based Deep NNs for Entity Discovery & Linking

  3. Overview What is FOFE? Entity Discovery methods FOFE-based Entity Linking Results and Conclusion What is FOFE? Definition (Fixed-size Ordinally Forgetting Encoding) S = w 1 , w 2 , ..., w n is a sequence of any discrete symbols; w i is represented as e i in 1-hot representation; the encoding of a partial sequence up to the t -th word is recursively defined as: � if t = 1 e t , z t = α · z t − 1 + e t , otherwise α ∈ (0 , 1) and t ∈ { Z | 1 ≤ x ≤ n } Nargiza Nosirova FOFE-based Deep NNs for Entity Discovery & Linking

  4. Overview What is FOFE? Entity Discovery methods FOFE-based Entity Linking Results and Conclusion What is FOFE? (continued) Example A = [1 , 0 , 0] B = [0 , 1 , 0] C = [0 , 0 , 1] ABC = [ α 2 , α, 1] Nargiza Nosirova FOFE-based Deep NNs for Entity Discovery & Linking

  5. Overview What is FOFE? Entity Discovery methods FOFE-based Entity Linking Results and Conclusion What is FOFE? (continued) Any variable length sequence is losslessly encoded into a fixed-size vector. WORD 1-HOT PARTIAL SEQUENCE FOFE 1000000 w 0 w 6 0 , 0 , 0 , 0 , 0 , 0 , 1 0100000 w 1 w 6 , w 4 0 , 0 , 0 , 0 , 1 , 0 , α w 2 0010000 0 , 0 , 0 , 0 , α, 1 , α 2 w 6 , w 4 , w 5 w 3 0001000 1 , 0 , 0 , 0 , α 2 , α, α 3 w 6 , w 4 , w 5 , w 0 w 4 0000100 α, 0 , 0 , 0 , α 3 , 1 + α 2 , α 4 w 6 , w 4 , w 5 , w 0 , w 5 0000010 w 5 α 2 , 0 , 0 , 0 , 1 + α 4 , α + α 3 , α 5 w 6 , w 4 , w 5 , w 0 , w 5 , w 4 w 6 0000001 Table: Partial encoding of w 6 , w 4 , w 5 , w 0 , w 5 , w 4 Table: Vocab of size 7 Nargiza Nosirova FOFE-based Deep NNs for Entity Discovery & Linking

  6. Overview What is FOFE? Entity Discovery methods FOFE-based Entity Linking Results and Conclusion Universal Framework for NLP Nargiza Nosirova FOFE-based Deep NNs for Entity Discovery & Linking Figure: FOFE-FFNN for NLP

  7. Overview What is FOFE? FOFE-based Model for Entity Discovery Entity Discovery methods Ensemble Modeling FOFE-based Entity Linking Multi-task Model for Entity Discovery using FOFE Results and Conclusion FOFE-based model for Entity Discovery Figure: Illustration of the local detection approach for Entity Discovery using FOFE codes as input and FFNN as model. Nargiza Nosirova FOFE-based Deep NNs for Entity Discovery & Linking

  8. Overview What is FOFE? FOFE-based Model for Entity Discovery Entity Discovery methods Ensemble Modeling FOFE-based Entity Linking Multi-task Model for Entity Discovery using FOFE Results and Conclusion FOFE-based Model for Entity Discovery (continued) Features used: Word-level: BOW vector of the fragment FOFE codes of the left/right contexts Character-level: FOFE code of the fragment Char CNN Nargiza Nosirova FOFE-based Deep NNs for Entity Discovery & Linking

  9. Overview What is FOFE? FOFE-based Model for Entity Discovery Entity Discovery methods Ensemble Modeling FOFE-based Entity Linking Multi-task Model for Entity Discovery using FOFE Results and Conclusion Ensemble Modeling 1 2 3 Nargiza Nosirova FOFE-based Deep NNs for Entity Discovery & Linking

  10. Overview What is FOFE? FOFE-based Model for Entity Discovery Entity Discovery methods Ensemble Modeling FOFE-based Entity Linking Multi-task Model for Entity Discovery using FOFE Results and Conclusion Multi-task Model for Entity Discovery using FOFE Multi-task Learning: Concurrently learning a task alongside related (auxiliary) tasks by using a shared representation. Word and character level features are also FOFE based. Make use of different datasets, each treated Figure: Illustration of the multi-task as separate tasks. FFNN approach for Entity Discovery. Nargiza Nosirova FOFE-based Deep NNs for Entity Discovery & Linking

  11. Overview What is FOFE? Entity Discovery methods FOFE-based Entity Linking Results and Conclusion FOFE-based Entity Linking 1 Rule-based candidate generation Generated based on knowledge bases (KB), such as Freebase, Wikipedia Outputs a candidate list (Freebase nodes) 2 Neural Network based probability ranking Candidate with the highest probability is chosen as the final linking result Nargiza Nosirova FOFE-based Deep NNs for Entity Discovery & Linking

  12. Overview What is FOFE? Entity Discovery methods FOFE-based Entity Linking Results and Conclusion FOFE-based Entity Linking (continued) Features used: FOFE codes of left/right context. BOW of mention. Mention’s candidates KB description as BOW normalized by tf-idf. Figure: Illustration of our FOFE-based Entity Linking system using FFNNs. Nargiza Nosirova FOFE-based Deep NNs for Entity Discovery & Linking

  13. Overview Results: Datasets What is FOFE? Ensemble Modeling Results Entity Discovery methods Multi-task Learning Results FOFE-based Entity Linking Entity Linking Results Results and Conclusion Conclusion Results: Datasets FOFE-based Entity Discovery and Entity Linking models Training data : KBP 2015 (train & eval), KBP 2016 (eval), and iFLYTEK’s in-house dataset. Multitask FOFE-based model Main task: KBP 2017 EDL task Auxiliary tasks: English: CoNLL-2003, OntoNotes 5.0 Spanish & Chinese: DEFT Light ERE dataset Nargiza Nosirova FOFE-based Deep NNs for Entity Discovery & Linking

  14. Overview Results: Datasets What is FOFE? Ensemble Modeling Results Entity Discovery methods Multi-task Learning Results FOFE-based Entity Linking Entity Linking Results Results and Conclusion Conclusion Ensemble Modeling Results LANG single model model ensemble P R F 1 P R F 1 ENG 0.801 0.745 0.772 0.808 0.774 0.791 CMN 0.775 0.660 0.713 0.793 0.726 0.758 SPA 0.856 0.715 0.779 0.839 0.773 0.805 ALL - - - 0.817 0.747 0.781 Table: Entity Discovery (ED) performance of model ensemble in the KBP 2017 trilingual EDL evaluation. Nargiza Nosirova FOFE-based Deep NNs for Entity Discovery & Linking

  15. Overview Results: Datasets What is FOFE? Ensemble Modeling Results Entity Discovery methods Multi-task Learning Results FOFE-based Entity Linking Entity Linking Results Results and Conclusion Conclusion Multi-task Learning Results LANG Single-task model Multi-task model P R F 1 P R F 1 ENG 0.866 0.706 0.778 0.878 0.705 0.782 CMN 0.795 0.635 0.707 0.789 0.665 0.722 SPA 0.919 0.631 0.748 0.844 0.738 0.787 ALL - - - 0.830 0.698 0.758 Table: Entity Discovery (ED) performance for multi-task learning in the KBP 2017 trilingual EDL evaluation. Nargiza Nosirova FOFE-based Deep NNs for Entity Discovery & Linking

  16. Overview Results: Datasets What is FOFE? Ensemble Modeling Results Entity Discovery methods Multi-task Learning Results FOFE-based Entity Linking Entity Linking Results Results and Conclusion Conclusion Entity Linking Results LANG baseline1 baseline2 FOFE-EL NERLC CEAFmC NERLC CEAFmC NERLC CEAFmC ENG 0.646 0.630 0.572 0.615 0.648 0.631 CMN 0.617 0.650 0.579 0.615 0.641 0.674 SPA 0.569 0.568 0.538 0.547 0.577 0.576 ALL 0.611 0.607 0.565 0.586 0.624 0.620 Table: Performance on the KBP2017 EDL evaluation of our three entity linking systems. ( NERLC denotes for strong typed all match and CEAFmC for typed mention ceaf ) Nargiza Nosirova FOFE-based Deep NNs for Entity Discovery & Linking

  17. Overview Results: Datasets What is FOFE? Ensemble Modeling Results Entity Discovery methods Multi-task Learning Results FOFE-based Entity Linking Entity Linking Results Results and Conclusion Conclusion Conclusion A local detection approach to Entity Discovery and MD by applying FFNN on top of FOFE An extended multi-task approach to Entity Discovery using FOFE No feature engineering and No external knowledge Strong results on the KBP 2017 EDL track Nargiza Nosirova FOFE-based Deep NNs for Entity Discovery & Linking

  18. Thank You! (Q&A) Nargiza Nosirova FOFE-based Deep NNs for Entity Discovery & Linking

Recommend

Learning Semantic Entity Representations with Knowledge Graph and Deep Neural Networks and its

Learning Semantic Entity Representations with Knowledge Graph and Deep Neural Networks and its

Learning Semantic Entity Representations with Knowledge Graph and Deep Neural Networks and its Application to Named Entity Disambiguation Hongzhao Huang 1 and Larry Heck 2 Computer Science Department, Rensselaer Polytechnic Institute 1 Microsoft

243 views • 23 slides
Exploring Neural Networks for Entity Discovery and Linking (EDL) Dan Liu 1 , Wei Lin 1 , Shiliang

Exploring Neural Networks for Entity Discovery and Linking (EDL) Dan Liu 1 , Wei Lin 1 , Shiliang

Exploring Neural Networks for Entity Discovery and Linking (EDL) Dan Liu 1 , Wei Lin 1 , Shiliang Zhang 2 , Si Wei 1 , Hui Jiang 3 1 i FLYTEK Research, Hefei, Anhui, China 2 University of Science and Technology of China, Hefei, Anhui, Mingbin Xu 3

673 views • 39 slides
Accelerating drug discovery with deep neural networks literature review Tobias Sikosek Senior

Accelerating drug discovery with deep neural networks literature review Tobias Sikosek Senior

Accelerating drug discovery with deep neural networks literature review Tobias Sikosek Senior Data Scientist In Silico Unit (Heidelberg) Deep Learning Artificial intelligence Machine learning Deep learning Renewed focus on multi-layer

836 views • 50 slides
Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan Zelener Machine Learning Reading Group January 7 th 2016 The Graduate Center, CUNY Objectives Explain some of the trends of deep learning and

680 views • 44 slides
Deep Neural Networks and Hidden Markov Models in i-vector-based Text-Dependent Speaker

Deep Neural Networks and Hidden Markov Models in i-vector-based Text-Dependent Speaker

Introduction HMM based method Deep Neural Networks (DNNs) Experiments and Results Conclusions Deep Neural Networks and Hidden Markov Models in i-vector-based Text-Dependent Speaker Verification Hossein Zeinali 1 , 2 , Luk s Burget 2 ,

301 views • 11 slides
Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In

987 views • 40 slides
Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information

Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information

Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information Engineering The Chinese University of Hong Kong Deep Neural Networks are Everywhere Playing Go Making Medical Decision Understanding Scenes Deep Neural

1.1k views • 46 slides
Neural Networks and Computation Graphs CS 6956: Deep Learning for NLP Based on slides and

Neural Networks and Computation Graphs CS 6956: Deep Learning for NLP Based on slides and

Neural Networks and Computation Graphs CS 6956: Deep Learning for NLP Based on slides and material from Geoffrey Hinton, Richard Socher, Yoav Goldberg, and others. The computation graph slides are based on a tutorial Practical Neural

1.5k views • 121 slides
On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya

On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya

On the Expressive Power of Deep Neural Networks Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, Jascha Sohl Dickstein Tom Brady Deep Neural Networks Recent successes in using Deep neural networks for image classification,

1.32k views • 17 slides
Outline Evolution of neurocomputing Artificial neural networks Feed forward

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides
Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova

Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova

Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova November 21, 2016 Neural Networks 2/20 Neural Networks 3/20 Neural Networks Neural computing requires a number of neurons , to be connected

813 views • 21 slides
Optimizing Deep Neural Networks Leena Chennuru Vankadara 26-10-2015 Table of Contents Neural

Optimizing Deep Neural Networks Leena Chennuru Vankadara 26-10-2015 Table of Contents Neural

Optimizing Deep Neural Networks Leena Chennuru Vankadara 26-10-2015 Table of Contents Neural Networks and loss surfaces Problems of Deep Architectures Optimization in Neural Networks Under fitting Proliferation of Saddle

1.67k views • 41 slides
Predicting Temporal Sets with Deep Neural Networks Le Yu, Leilei Sun*, Bowen Du, Chuanren Liu, Hui

Predicting Temporal Sets with Deep Neural Networks Le Yu, Leilei Sun*, Bowen Du, Chuanren Liu, Hui

Proceedings of the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 20), August 23 27, 2020, Virtual Event, CA, USA. Predicting Temporal Sets with Deep Neural Networks Le Yu, Leilei Sun*, Bowen Du, Chuanren Liu, Hui

440 views • 18 slides
Deep Neural Networks and Molecular Dynamics Roberto Car Princeton University MAX Conference

Deep Neural Networks and Molecular Dynamics Roberto Car Princeton University MAX Conference

Deep Neural Networks and Molecular Dynamics Roberto Car Princeton University MAX Conference ICTP, Trieste, Jan 29-31 2018 Deep Neural Networks When properly trained, deep NN are capable of learning the complex functional dependence of

442 views • 17 slides
CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided Backpropagation, Deep Dream, Deep Art, Fooling Convolutional Neural Networks Mitesh M. Khapra Department of Computer Science and Engineering Indian Institute

1.07k views • 72 slides
Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and Courville [chapt. 6,7,8]; AIMA [sect. 21.1-21.3]; Sutton and Barto, Reinforcement Learning: an

527 views • 35 slides
Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Learning in Neural Networks w 1,3 w 3,5 1 3 5 w w 1,4 3,6 w w 2,3 4,5 2 4 6 w w 2,4 4,6 Class #18: Back-Propagation; } A neural network can learn a classification function by Tuning Hyper-Parameters adjusting its weights to

259 views • 4 slides
Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the

Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the

Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream Umut Gcl and Marcel A. J. van Gerven Article overview by Ilya Kuzovkin Computational Neuroscience Seminar University of Tartu

852 views • 63 slides
Recurrent Neural Networks Deep neural networks have enabled major advances in machine learning

Recurrent Neural Networks Deep neural networks have enabled major advances in machine learning

Recurrent Neural Networks Deep neural networks have enabled major advances in machine learning and AI y t-1 y t y t+1 Computer vision h t-1 h t h t+1 Language translation Speech recognition h t-1 h t h t+1 Question answering x t-1 x t x t+1

454 views • 43 slides
Concise Introduction to Deep Neural Networks Outline: Classification problems Motivating

Concise Introduction to Deep Neural Networks Outline: Classification problems Motivating

Concise Introduction to Deep Neural Networks Outline: Classification problems Motivating Deep (large) Neural Network (DNN) classifiers Neurons and DNN architectures Numerical training of DNNs (supervised deep learning) Spiking

630 views • 33 slides
ICASSP 2017 Tutorial on Methods for Interpreting and Understanding Deep Neural Networks G.

ICASSP 2017 Tutorial on Methods for Interpreting and Understanding Deep Neural Networks G.

ICASSP 2017 Tutorial on Methods for Interpreting and Understanding Deep Neural Networks G. Montavon, W. Samek, K.-R. Mller Part 2: Making Deep Neural Networks Transparent 5 March 2017 Making Deep Neural Nets Transparent DNN transparency

448 views • 44 slides
Deep Neural Networks CMSC 422 M ARINE C ARPUAT marine@cs.umd.edu Deep learning slides credit:

Deep Neural Networks CMSC 422 M ARINE C ARPUAT marine@cs.umd.edu Deep learning slides credit:

Deep Neural Networks CMSC 422 M ARINE C ARPUAT marine@cs.umd.edu Deep learning slides credit: Vlad Morariu Training (Deep) Neural Networks Computational graphs Improvements to gradient descent Stochastic gradient descent Momentum

814 views • 21 slides
Exploring Bit-Slice Sparsity in Deep Neural Networks for Efficient ReRAM-Based Deployment

Exploring Bit-Slice Sparsity in Deep Neural Networks for Efficient ReRAM-Based Deployment

Exploring Bit-Slice Sparsity in Deep Neural Networks for Efficient ReRAM-Based Deployment Jingyang Zhang 1 , Huanrui Yang 1 , Fan Chen 1 , Yitu Wang 2 , Hai Li 1 1 Duke University, 2 Fudan University EMC2 Workshop @ NeurIPS 2019 Motivation:

369 views • 6 slides
6. Convolutional Neural Networks CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from

6. Convolutional Neural Networks CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from

6. Convolutional Neural Networks CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from Zsolt Kira Quiz coming up Next Monday (2/5) 30 minutes Topics: Optimization Basic neural networks Neural Network

619 views • 41 slides

More recommend