Computational linguistics and NLP: How far from generic linguistics? - PowerPoint PPT Presentation

Is it linguistics at all? Differences from ‘traditional’ or ‘generic’ linguistics ◮ Traditional linguistics usually describes and compares languages. ◮ NLP is closer to mathematics and engineering: we calculate. ◮ Building computational models of linguistic phenomena: 1. ‘rule-based’ (‘hand-crafted’); 2. ‘data-driven’ (statistical). ◮ Statistics is at the core of today’s NLP . 5

Is it linguistics at all? Differences from ‘traditional’ or ‘generic’ linguistics ◮ Traditional linguistics usually describes and compares languages. ◮ NLP is closer to mathematics and engineering: we calculate. ◮ Building computational models of linguistic phenomena: 1. ‘rule-based’ (‘hand-crafted’); 2. ‘data-driven’ (statistical). ◮ Statistics is at the core of today’s NLP . ◮ We run experiments to test hypotheses: 5

Is it linguistics at all? Differences from ‘traditional’ or ‘generic’ linguistics ◮ Traditional linguistics usually describes and compares languages. ◮ NLP is closer to mathematics and engineering: we calculate. ◮ Building computational models of linguistic phenomena: 1. ‘rule-based’ (‘hand-crafted’); 2. ‘data-driven’ (statistical). ◮ Statistics is at the core of today’s NLP . ◮ We run experiments to test hypotheses: ◮ ‘ there are 10 parts of speech in this language ’, 5

Is it linguistics at all? Differences from ‘traditional’ or ‘generic’ linguistics ◮ Traditional linguistics usually describes and compares languages. ◮ NLP is closer to mathematics and engineering: we calculate. ◮ Building computational models of linguistic phenomena: 1. ‘rule-based’ (‘hand-crafted’); 2. ‘data-driven’ (statistical). ◮ Statistics is at the core of today’s NLP . ◮ We run experiments to test hypotheses: ◮ ‘ there are 10 parts of speech in this language ’, ◮ ‘ word co-occurrence information improves document classification ’. 5

Is it linguistics at all? Differences from ‘traditional’ or ‘generic’ linguistics ◮ Traditional linguistics usually describes and compares languages. ◮ NLP is closer to mathematics and engineering: we calculate. ◮ Building computational models of linguistic phenomena: 1. ‘rule-based’ (‘hand-crafted’); 2. ‘data-driven’ (statistical). ◮ Statistics is at the core of today’s NLP . ◮ We run experiments to test hypotheses: ◮ ‘ there are 10 parts of speech in this language ’, ◮ ‘ word co-occurrence information improves document classification ’. ◮ Replicability (the same experiment must always yield the same result); 5

Is it linguistics at all? Differences from ‘traditional’ or ‘generic’ linguistics ◮ Traditional linguistics usually describes and compares languages. ◮ NLP is closer to mathematics and engineering: we calculate. ◮ Building computational models of linguistic phenomena: 1. ‘rule-based’ (‘hand-crafted’); 2. ‘data-driven’ (statistical). ◮ Statistics is at the core of today’s NLP . ◮ We run experiments to test hypotheses: ◮ ‘ there are 10 parts of speech in this language ’, ◮ ‘ word co-occurrence information improves document classification ’. ◮ Replicability (the same experiment must always yield the same result); ◮ Reproducibility (similar experiments should yield comparable results). 5

Stress on practice 6

Stress on practice ◮ Research should be practical. 6

Stress on practice ◮ Research should be practical. ◮ ‘ Show me your code! ’ 6

Stress on practice ◮ Research should be practical. ◮ ‘ Show me your code! ’ ◮ ‘ Show me the scores of your system! ’ 6

Stress on practice ◮ Research should be practical. ◮ ‘ Show me your code! ’ ◮ ‘ Show me the scores of your system! ’ ◮ Empirical evaluation on particular problems. 6

Stress on practice ◮ Research should be practical. ◮ ‘ Show me your code! ’ ◮ ‘ Show me the scores of your system! ’ ◮ Empirical evaluation on particular problems. ◮ Test data sets. 6

Stress on practice ◮ Research should be practical. ◮ ‘ Show me your code! ’ ◮ ‘ Show me the scores of your system! ’ ◮ Empirical evaluation on particular problems. ◮ Test data sets. ◮ Shared tasks (competitions). 6

Publishing activities ◮ Conferences: ◮ ACL 7

Publishing activities ◮ Conferences: ◮ ACL ◮ EMNLP 7

Publishing activities ◮ Conferences: ◮ ACL ◮ EMNLP ◮ EACL 7

Publishing activities ◮ Conferences: ◮ ACL ◮ EMNLP ◮ EACL ◮ NAACL 7

Publishing activities ◮ Conferences: ◮ ACL ◮ EMNLP ◮ EACL ◮ NAACL ◮ COLING 7

Publishing activities ◮ Conferences: ◮ ACL ◮ EMNLP ◮ EACL ◮ NAACL ◮ COLING ◮ LREC... ◮ Journals: 7

Publishing activities ◮ Conferences: ◮ ACL ◮ EMNLP ◮ EACL ◮ NAACL ◮ COLING ◮ LREC... ◮ Journals: ◮ ‘ Computational Linguistics ’ (CL); 7

Publishing activities ◮ Conferences: ◮ ACL ◮ EMNLP ◮ EACL ◮ NAACL ◮ COLING ◮ LREC... ◮ Journals: ◮ ‘ Computational Linguistics ’ (CL); ◮ ‘ Transactions of the Association for Computational Linguistics ’ (TACL). 7

Publishing activities ◮ Conferences: ◮ ACL ◮ EMNLP ◮ EACL ◮ NAACL ◮ COLING ◮ LREC... ◮ Journals: ◮ ‘ Computational Linguistics ’ (CL); ◮ ‘ Transactions of the Association for Computational Linguistics ’ (TACL). ◮ Unlike in other fields, journals are not that important. 7

Publishing activities ◮ Conferences: ◮ ACL ◮ EMNLP ◮ EACL ◮ NAACL ◮ COLING ◮ LREC... ◮ Journals: ◮ ‘ Computational Linguistics ’ (CL); ◮ ‘ Transactions of the Association for Computational Linguistics ’ (TACL). ◮ Unlike in other fields, journals are not that important. 7

❤tt♣s✿✴✴❛r①✐✈✳♦r❣✴❧✐st✴❝s✳❈▲✴r❡❝❡♥t Publishing activities ◮ Most of the papers can be found in the Association for Computational Linguistics (ACL) Anthology: ◮ ❤tt♣s✿✴✴❛❝❧❛♥t❤♦❧♦❣②✳✐♥❢♦✴ 8

❤tt♣s✿✴✴❛r①✐✈✳♦r❣✴❧✐st✴❝s✳❈▲✴r❡❝❡♥t Publishing activities ◮ Most of the papers can be found in the Association for Computational Linguistics (ACL) Anthology: ◮ ❤tt♣s✿✴✴❛❝❧❛♥t❤♦❧♦❣②✳✐♥❢♦✴ ◮ Double blind peer review almost everywhere... 8

❤tt♣s✿✴✴❛r①✐✈✳♦r❣✴❧✐st✴❝s✳❈▲✴r❡❝❡♥t Publishing activities ◮ Most of the papers can be found in the Association for Computational Linguistics (ACL) Anthology: ◮ ❤tt♣s✿✴✴❛❝❧❛♥t❤♦❧♦❣②✳✐♥❢♦✴ ◮ Double blind peer review almost everywhere... ◮ ...recent years: open preprints published online: 8

Publishing activities ◮ Most of the papers can be found in the Association for Computational Linguistics (ACL) Anthology: ◮ ❤tt♣s✿✴✴❛❝❧❛♥t❤♦❧♦❣②✳✐♥❢♦✴ ◮ Double blind peer review almost everywhere... ◮ ...recent years: open preprints published online: ◮ ❤tt♣s✿✴✴❛r①✐✈✳♦r❣✴❧✐st✴❝s✳❈▲✴r❡❝❡♥t 8

Publishing activities ◮ Most of the papers can be found in the Association for Computational Linguistics (ACL) Anthology: ◮ ❤tt♣s✿✴✴❛❝❧❛♥t❤♦❧♦❣②✳✐♥❢♦✴ ◮ Double blind peer review almost everywhere... ◮ ...recent years: open preprints published online: ◮ ❤tt♣s✿✴✴❛r①✐✈✳♦r❣✴❧✐st✴❝s✳❈▲✴r❡❝❡♥t 8

Machine learning ◮ NLP is now being rapidly transformed by another field(s): 9

Machine learning ◮ NLP is now being rapidly transformed by another field(s): ◮ data science and machine learning. 9

Machine learning ◮ NLP is now being rapidly transformed by another field(s): ◮ data science and machine learning. ◮ Some problems are so complex that we can‘t formulate exact algorithms for them. 9

Machine learning ◮ NLP is now being rapidly transformed by another field(s): ◮ data science and machine learning. ◮ Some problems are so complex that we can‘t formulate exact algorithms for them. ◮ To solve such problems, one can use machine learning: 9

Machine learning ◮ NLP is now being rapidly transformed by another field(s): ◮ data science and machine learning. ◮ Some problems are so complex that we can‘t formulate exact algorithms for them. ◮ To solve such problems, one can use machine learning: ◮ programs which learn to make correct decisions on some training material and improve with experience; 9

Machine learning ◮ NLP is now being rapidly transformed by another field(s): ◮ data science and machine learning. ◮ Some problems are so complex that we can‘t formulate exact algorithms for them. ◮ To solve such problems, one can use machine learning: ◮ programs which learn to make correct decisions on some training material and improve with experience; ◮ thus, we train our systems on linguistic data (usually large text collections: corpora). 9

Machine learning ◮ NLP is now being rapidly transformed by another field(s): ◮ data science and machine learning. ◮ Some problems are so complex that we can‘t formulate exact algorithms for them. ◮ To solve such problems, one can use machine learning: ◮ programs which learn to make correct decisions on some training material and improve with experience; ◮ thus, we train our systems on linguistic data (usually large text collections: corpora). ◮ Artificial neural networks are one of popular machine learning approaches for language modeling. 9

Machine learning ◮ NLP is now being rapidly transformed by another field(s): ◮ data science and machine learning. ◮ Some problems are so complex that we can‘t formulate exact algorithms for them. ◮ To solve such problems, one can use machine learning: ◮ programs which learn to make correct decisions on some training material and improve with experience; ◮ thus, we train our systems on linguistic data (usually large text collections: corpora). ◮ Artificial neural networks are one of popular machine learning approaches for language modeling. Deep learning renaissance ◮ ‘Deep learning’ is training and using multi-layered artificial neural networks. 9

Machine learning ◮ NLP is now being rapidly transformed by another field(s): ◮ data science and machine learning. ◮ Some problems are so complex that we can‘t formulate exact algorithms for them. ◮ To solve such problems, one can use machine learning: ◮ programs which learn to make correct decisions on some training material and improve with experience; ◮ thus, we train our systems on linguistic data (usually large text collections: corpora). ◮ Artificial neural networks are one of popular machine learning approaches for language modeling. Deep learning renaissance ◮ ‘Deep learning’ is training and using multi-layered artificial neural networks. ◮ After long ‘winter’ (since 60s and 70s), it is now again popular. 9

Machine learning ◮ NLP is now being rapidly transformed by another field(s): ◮ data science and machine learning. ◮ Some problems are so complex that we can‘t formulate exact algorithms for them. ◮ To solve such problems, one can use machine learning: ◮ programs which learn to make correct decisions on some training material and improve with experience; ◮ thus, we train our systems on linguistic data (usually large text collections: corpora). ◮ Artificial neural networks are one of popular machine learning approaches for language modeling. Deep learning renaissance ◮ ‘Deep learning’ is training and using multi-layered artificial neural networks. ◮ After long ‘winter’ (since 60s and 70s), it is now again popular. ◮ Deep neural approaches are very efficient in NLP . 9

Machine learning ◮ NLP is now being rapidly transformed by another field(s): ◮ data science and machine learning. ◮ Some problems are so complex that we can‘t formulate exact algorithms for them. ◮ To solve such problems, one can use machine learning: ◮ programs which learn to make correct decisions on some training material and improve with experience; ◮ thus, we train our systems on linguistic data (usually large text collections: corpora). ◮ Artificial neural networks are one of popular machine learning approaches for language modeling. Deep learning renaissance ◮ ‘Deep learning’ is training and using multi-layered artificial neural networks. ◮ After long ‘winter’ (since 60s and 70s), it is now again popular. ◮ Deep neural approaches are very efficient in NLP . ◮ ‘ Do we need anything except neural networks now? ’ 9

Machine learning ◮ NLP is now being rapidly transformed by another field(s): ◮ data science and machine learning. ◮ Some problems are so complex that we can‘t formulate exact algorithms for them. ◮ To solve such problems, one can use machine learning: ◮ programs which learn to make correct decisions on some training material and improve with experience; ◮ thus, we train our systems on linguistic data (usually large text collections: corpora). ◮ Artificial neural networks are one of popular machine learning approaches for language modeling. Deep learning renaissance ◮ ‘Deep learning’ is training and using multi-layered artificial neural networks. ◮ After long ‘winter’ (since 60s and 70s), it is now again popular. ◮ Deep neural approaches are very efficient in NLP . ◮ ‘ Do we need anything except neural networks now? ’ ◮ Another reason for the recent boost of interest towards our discipline. 9

Problems and challenges NLP has its problems 10

Problems and challenges NLP has its problems ◮ equity and diversity; 10

Problems and challenges NLP has its problems ◮ equity and diversity; ◮ traditional reviewing schemes conflicting with ArXiv: 10

Problems and challenges NLP has its problems ◮ equity and diversity; ◮ traditional reviewing schemes conflicting with ArXiv: ◮ how to preserve anonymity? 10

Problems and challenges NLP has its problems ◮ equity and diversity; ◮ traditional reviewing schemes conflicting with ArXiv: ◮ how to preserve anonymity? ◮ preprint publishing is good in disseminating science and making it open, but... 10

Problems and challenges NLP has its problems ◮ equity and diversity; ◮ traditional reviewing schemes conflicting with ArXiv: ◮ how to preserve anonymity? ◮ preprint publishing is good in disseminating science and making it open, but... ◮ ...people can use ArXiv for flag-planting, and to simply circumvent the peer-review process. 10

Problems and challenges NLP has its problems ◮ equity and diversity; ◮ traditional reviewing schemes conflicting with ArXiv: ◮ how to preserve anonymity? ◮ preprint publishing is good in disseminating science and making it open, but... ◮ ...people can use ArXiv for flag-planting, and to simply circumvent the peer-review process. ◮ machine learning models amplifying biases and discrimination in data [Zhao et al., 2017] 10

Problems and challenges NLP has its problems ◮ equity and diversity; ◮ traditional reviewing schemes conflicting with ArXiv: ◮ how to preserve anonymity? ◮ preprint publishing is good in disseminating science and making it open, but... ◮ ...people can use ArXiv for flag-planting, and to simply circumvent the peer-review process. ◮ machine learning models amplifying biases and discrimination in data [Zhao et al., 2017] ◮ sometimes research success depends on computational power: ◮ ‘ ...do we have enough GPUs? ’ 10

Science? ◮ People wonder: ◮ ‘ What are your research questions? ’ ◮ ‘ Just lots of numbers with very small differences? ’ 11

Science? ◮ People wonder: ◮ ‘ What are your research questions? ’ ◮ ‘ Just lots of numbers with very small differences? ’ ◮ Is it a science or an engineering discipline? 11

Science? ◮ People wonder: ◮ ‘ What are your research questions? ’ ◮ ‘ Just lots of numbers with very small differences? ’ ◮ Is it a science or an engineering discipline? ◮ Or may be CL is a science and NLP is its application towards empirical problems? 11

Science? ◮ People wonder: ◮ ‘ What are your research questions? ’ ◮ ‘ Just lots of numbers with very small differences? ’ ◮ Is it a science or an engineering discipline? ◮ Or may be CL is a science and NLP is its application towards empirical problems? ◮ Motivation for research can be different: 11

Science? ◮ People wonder: ◮ ‘ What are your research questions? ’ ◮ ‘ Just lots of numbers with very small differences? ’ ◮ Is it a science or an engineering discipline? ◮ Or may be CL is a science and NLP is its application towards empirical problems? ◮ Motivation for research can be different: 1. trying to provide a computational explanation for linguistic or psycholinguistic phenomenon; 11

Science? ◮ People wonder: ◮ ‘ What are your research questions? ’ ◮ ‘ Just lots of numbers with very small differences? ’ ◮ Is it a science or an engineering discipline? ◮ Or may be CL is a science and NLP is its application towards empirical problems? ◮ Motivation for research can be different: 1. trying to provide a computational explanation for linguistic or psycholinguistic phenomenon; 2. trying to provide a working component of a speech or natural language system. 11

Science? ◮ People wonder: ◮ ‘ What are your research questions? ’ ◮ ‘ Just lots of numbers with very small differences? ’ ◮ Is it a science or an engineering discipline? ◮ Or may be CL is a science and NLP is its application towards empirical problems? ◮ Motivation for research can be different: 1. trying to provide a computational explanation for linguistic or psycholinguistic phenomenon; 2. trying to provide a working component of a speech or natural language system. ◮ Do our top-tier conferences belong to CL or to NLP then? 11

Science? ◮ People wonder: ◮ ‘ What are your research questions? ’ ◮ ‘ Just lots of numbers with very small differences? ’ ◮ Is it a science or an engineering discipline? ◮ Or may be CL is a science and NLP is its application towards empirical problems? ◮ Motivation for research can be different: 1. trying to provide a computational explanation for linguistic or psycholinguistic phenomenon; 2. trying to provide a working component of a speech or natural language system. ◮ Do our top-tier conferences belong to CL or to NLP then? ◮ The overwhelming majority of papers are empirical today. 11

Science? ◮ People wonder: ◮ ‘ What are your research questions? ’ ◮ ‘ Just lots of numbers with very small differences? ’ ◮ Is it a science or an engineering discipline? ◮ Or may be CL is a science and NLP is its application towards empirical problems? ◮ Motivation for research can be different: 1. trying to provide a computational explanation for linguistic or psycholinguistic phenomenon; 2. trying to provide a working component of a speech or natural language system. ◮ Do our top-tier conferences belong to CL or to NLP then? ◮ The overwhelming majority of papers are empirical today. ◮ No final answer yet. 11

Language IS complicated 12

Language IS complicated ‘...human language is magnificent, and complex, and challenging. It has tons of nuances, and corners, and oddities, and surprises. 12

Language IS complicated ‘...human language is magnificent, and complex, and challenging. It has tons of nuances, and corners, and oddities, and surprises. While natural language processing researchers, and natural language generation researchers—and linguists! who do a lot of the heavy lifting—made some impressive advances towards our understanding of language and how to process it, we are still just barely scratching the surface on this.’ 12

Interaction with traditional linguistics Linguistics is back 13

Interaction with traditional linguistics Linguistics is back ◮ NLP is re-embracing linguistic structure now; 13

Interaction with traditional linguistics Linguistics is back ◮ NLP is re-embracing linguistic structure now; ◮ Even the strongest proponents of purely data-driven approaches acknowledge it; 13

Interaction with traditional linguistics Linguistics is back ◮ NLP is re-embracing linguistic structure now; ◮ Even the strongest proponents of purely data-driven approaches acknowledge it; ◮ Linguistic structures induced into machine learning systems reduce search space, bringing improvements [Dyer, 2017] ; 13

Interaction with traditional linguistics Linguistics is back ◮ NLP is re-embracing linguistic structure now; ◮ Even the strongest proponents of purely data-driven approaches acknowledge it; ◮ Linguistic structures induced into machine learning systems reduce search space, bringing improvements [Dyer, 2017] ; ◮ Language is not just sequences of words / characters / bytes. 13

Interaction with traditional linguistics Linguistics is back ◮ NLP is re-embracing linguistic structure now; ◮ Even the strongest proponents of purely data-driven approaches acknowledge it; ◮ Linguistic structures induced into machine learning systems reduce search space, bringing improvements [Dyer, 2017] ; ◮ Language is not just sequences of words / characters / bytes. ◮ But what can NLP give to traditional linguistics? 13

Interaction with traditional linguistics Linguistics is back ◮ NLP is re-embracing linguistic structure now; ◮ Even the strongest proponents of purely data-driven approaches acknowledge it; ◮ Linguistic structures induced into machine learning systems reduce search space, bringing improvements [Dyer, 2017] ; ◮ Language is not just sequences of words / characters / bytes. ◮ But what can NLP give to traditional linguistics? ◮ Or to humanities in general? 13

Interaction with traditional linguistics Linguistics is back ◮ NLP is re-embracing linguistic structure now; ◮ Even the strongest proponents of purely data-driven approaches acknowledge it; ◮ Linguistic structures induced into machine learning systems reduce search space, bringing improvements [Dyer, 2017] ; ◮ Language is not just sequences of words / characters / bytes. ◮ But what can NLP give to traditional linguistics? ◮ Or to humanities in general? ◮ I will now outline 2 case studies from my own research. 13

Contents What is NLP? 1 Case 1: Redefining parts of speech 2 Case 2: Tracing diachronic semantic shifts 3 13

‘Redefining parts of speech with word embeddings’ (Presented at the CoNLL2016, [Kutuzov et al., 2016] ) 14