Representing symbolic linguistic structures for neural NLP: methods - PowerPoint PPT Presentation

Representing symbolic linguistic structures for neural NLP: methods and applications Alexander Panchenko Assistant Professor for NLP

Structure and goals of this talk • Publishing in ACL and similar conferences, e.g. NAACL, EMNLP , CoNLL: • this is the top conference in NLP —> your work is visible • Topics of this talk (all are based on forthcoming publications at ACL’19 and associated workshops): • Encoding and using linguistic structures in neural NLP models • Argument Mining

About myself: a decade of fun R&D in NLP • 2019-now: Skoltech , Assistant Professor in NLP , MOSCOW • 2017-2019: University of Hamburg , Postdoc in NLP , GERMANY • 2015-2017: TU Darmstadt , Postdoc in NLP , GERMANY • 2013-2015: Startup in SNA , Research Engineer in NLP , MOSCOW • 2009-2013: Université catholique de Louvain , PhD in Computational Linguistics, BELGIUM • 2008: Xerox Research Centre Europe , Research Intern, FRANCE • 2002-2008: Bauman Moscow State Technical University , Engineer in Information Systems, MOSCOW

About myself: a decade of fun R&D in NLP • Publications in int’l conferences & journals: • ACL • EMNLP • EACL • ECIR • NLE • Best papers at Representation learning workshop (ACL’2016) and SemEval’2019. • Editor and co-chair : • Cambridge Natural Language Engineering (NLE) • Springer LNCS/CCIS: AIST conf. • PC : • ACL, NAACL, EMNLP , CoNLL, LREC, RANLP , COLING, …

How to inform neural architectures for NLP with symbolic linguistic knowledge? • Special issue of the Natural Language Engineering journal on informing neural architectures for NLP with linguistic and background knowledge: https://sites.google.com/view/nlesi

How to inform neural architectures for NLP with symbolic linguistic knowledge? Some options : • Graph embeddings • Poincaré embeddings • Regularisers that access the resource • Structure of neural network is based on the structure of the resource • … other specialised embeddings? • … invented by you? • Special issue of the Natural Language Engineering journal on informing neural architectures for NLP with linguistic and background knowledge: https://sites.google.com/view/nlesi

Text: a sparse symbolic representation Image source: https://www.tensorflow.org/tutorials/word2vec

Graph: a sparse symbolic representation

Embedding graph into a vector space From a survey on graph embeddings [Hamilton et al., 2017]:

Learning with an autoencoder From a survey on graph embeddings [Hamilton et al., 2017]:

A summary of well-known graph embedding algorithms From a survey on graph embeddings [Hamilton et al., 2017]:

Graph Metric Embeddings • A short paper at ACL 2019 • Paper : https://arxiv.org/abs/ 1906.07040 • Code : http://github.com/uhh- lt/path2vec

path2vec model

Computational gains compare to graph-based algorithms Similarity computation: graph vs vectors

path2vec: evaluation results on three different graphs Evaluation on di ff erent graphs on SimLex999 (left) and shortest path distance (middle, right).

path2vec evaluation inside a graph- based WSD algorithm (WordNet graph)

Graph embeddings for neural entity linking • A short paper at ACL 2019 Student Research Workshop (main conference) • Paper: https://www.inf.uni- hamburg.de/en/inst/ab/lt/ publications/2019-sevgilietal- aclsrw-graphemb.pdf • Code : https://github.com/ uhh-lt/kb2vec

What is Entity Linking? Source of image: https://medium.com/asgard-ai/how-to-enhance-automatic-text-analysis- with-entity-linking-29128a12b

Challenges of Entity Linking Michael Jordan (NBA) vs Michael Jordan (LDA), etc. Ambiguity ruin everything: Source of image: https://medium.com/asgard-ai/how-to-enhance-automatic-text-analysis- with-entity-linking-29128a12b

Graph embeddings for neural entity linking

Graph embeddings for neural entity linking Architecture of our feed-forward neural ED system: using Wikipedia hyperlink graph embeddings as an additional input representation of entity candidates

Graph embeddings for neural entity linking

end2end Entity Linking Model by Kolistas et al. (2018) • The final score is used for both the mention linking and entity disambiguation decisions. • SOTA entity linking results.

Graph embeddings for neural entity linking

Poincaré embeddings for various NLP tasks • ACL 2019 full paper • Paper : https://www.inf.uni- hamburg.de/en/inst/ab/lt/ publications/2019-janaetal- aclmain-poincompo.pdf • Code : https://github.com/ uhh-lt/poincare

Poincaré embeddings for various NLP tasks Contributions: • We devise a straightforward and e ffi cient approach for combining distributional and hypernymy information for the task of noun phrase compositionality prediction . As far as we are aware, this is the first application of Poincaré embeddings to this task. • We demonstrate consistent and significant improvements on benchmark datasets in un- supervised and supervised settings.

Poincaré embeddings for various NLP tasks Image source: • Poincaré ball: https://arxiv.org/pdf/1705.08039.pdf • Distance on a ball between two points: •

Poincaré embeddings for various NLP tasks Training objective: Training data: • A set of relations (apple IsA fruit) • Can be taken from WordNet • … or extracted from text Source of the image: https://arxiv.org/pdf/1902.00913.pdf

Poincaré embeddings for noun compositionally hot dog —> food BUT dog —> animal green apple —> fruit AND apple —> fruit Evaluation results: comparison to the distributional models

Noun compositionality for the Russian language • The Balto-Slavic NLP workshop at ACL 2019 • Paper: http://panchenko.me/ papers/bsnlp19.pdf • Code: https://github.com/ slangtech/ru-comps • A dataset for evaluation of noun compositionally for Russian.

Noun Compositionality for Russian: Results

Poincaré embeddings for taxonomy induction • A short paper at ACL 2019 • Paper : https://www.inf.uni- hamburg.de/en/inst/ab/lt/ publications/2019-alyetal- aclshort-hypertaxi.pdf • Code : https://github.com/ uhh-lt/ Taxonomy_Refinement_Embe ddings

Abandoned children in a taxonomy problem Attaching unconnected nodes in taxonomy provides large boosts in performance:

Poincaré embeddings for taxonomy induction Outline of our taxonomy refinement method:

Poincaré embeddings for taxonomy induction

Comparative Argument Mining • 6th Workshop on Argument Mining at ACL 2019 . • Paper : https://www.inf.uni- hamburg.de/en/inst/ab/lt/ publications/2019-panchenkoetal- argminingws-compsent.pdf • Code : https://github.com/uhh-lt/ comparative

Comparative Argument Mining • Sentiment analysis ++ • … not only opinions but also objective arguments. • … from text only. • Retrieve pros and cons to make some informed decisions.

Comparative Argument Mining • Sentiment analysis ++ • … not only opinions but also objective arguments. • … from text only. • Retrieve pros and cons to make some informed decisions.

Categorizing Comparative Sentences Contributions: • We release a new dataset consisting of 7,199 sentences containing item pairs (27% of the sentences are tagged as comparative and annotated with a preference); • We present an experimental study of supervised classifiers and a strong rule-based baseline from prior work.

Categorizing Comparative Sentences

Categorizing Comparative Sentences

Categorizing Comparative Sentences

Argument Mining Demo • Demo paper at ACL 2019 • Paper : https://www.inf.uni- hamburg.de/en/inst/ab/lt/ publications/2019-chernodubetal- acl19demo-targer.pdf • Code : • A neural sequence tagger: https://github.com/uhh-lt/targer • A web application for AM: http:// github.com/achernodub/targer/ • Demo : http:// ltdemos.informatik.uni- hamburg.de/targer/

Argument Mining Demo Tagger: • BiLSTM-CNN-CRF • A custom PyTorch implementation • CoNLL as input: can be easily used for any sequence labelling task

Argument Mining Demo Analyze Text: input field, drop-down model selection, colorised labels, and tagged result. http://ltdemos.informatik.uni-hamburg.de/targer/

Argument Mining Demo Search Arguments: query box, field selectors, and result with link to the original document. http://ltdemos.informatik.uni-hamburg.de/targer/

Argument Mining Demo

How to publish NLP research in conferences like *ACL? Select a relevant topic: • “Hot topic” may boost interest to your work • … but many people may be working on it at the same time (you need to be fast). • … especially if the idea is fairly straightforward extension of existing stu ff . Collaborate: • Find strong collaborators which already published in the conferences you are aiming at. • Ideally your competences should complement one another. • Splitting work into “experiments”, “writing”, “related work”, etc.