computational linguistics hci evaluation measures
play

Computational Linguistics: HCI, Evaluation Measures R AFFAELLA B - PowerPoint PPT Presentation

Jan 16, 2023 •36 likes •856 views

Computational Linguistics: HCI, Evaluation Measures R AFFAELLA B ERNARDI U NIVERSIT ` A DEGLI S TUDI DI T RENTO E - MAIL : BERNARDI @ DISI . UNITN . IT Contents First Last Prev Next Contents 1 Today main topics . . . . . . . . . . . .

  1. Computational Linguistics: HCI, Evaluation Measures R AFFAELLA B ERNARDI U NIVERSIT ` A DEGLI S TUDI DI T RENTO E - MAIL : BERNARDI @ DISI . UNITN . IT Contents First Last Prev Next ◭

  2. Contents 1 Today main topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2 HCI via Natural Language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 3 Natural Language Interfaces to Data Bases . . . . . . . . . . . . . . . . . . . . . . . . 9 3.1 Sample Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.2 Restricted NL input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 4 From the ’90 .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 5 Question Answering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 5.1 History of ideas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 5.2 Re-emergence of QA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 5.3 Sample of QA architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 5.3.1 Question Classification: Why and How? . . . . . . . . . . . 18 5.3.2 Question Classification: Problems . . . . . . . . . . . . . . . . 19 5.3.3 Document Retrieval in QA . . . . . . . . . . . . . . . . . . . . . . 20 5.3.4 Document Retrieval in QA: Approaches . . . . . . . . . . . 21 5.3.5 Candidate Answer Extraction . . . . . . . . . . . . . . . . . . . . 22 5.3.6 Answer Re-ranking and Selection . . . . . . . . . . . . . . . . 23 5.4 Enriched QA Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Contents First Last Prev Next ◭

  3. 5.4.1 Fact Caching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 5.4.2 Other QA issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 5.5 Understanding in QA?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 5.6 Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 5.7 Today research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 6 Interactive Question Answering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 6.1 Characteristic of IQA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 6.2 Context in IQA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 6.3 Context in Open Domain IQA . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 6.4 System-initiated Interactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 6.4.1 Type of system initiatives . . . . . . . . . . . . . . . . . . . . . . . 35 6.4.2 System-initiated contributions . . . . . . . . . . . . . . . . . . . 36 6.5 System initiative in DB IQA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 6.6 Ontology in IQA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 6.7 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 6.8 Characteristics for the Ontology . . . . . . . . . . . . . . . . . . . . . . . . . . 40 6.9 Today research on Dialogues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 7.1 New trend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Contents First Last Prev Next ◭

  4. 7.2 Evaluation campaigns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 8 Information Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 8.1 History of IR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 8.2 IR challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 8.3 Retrieval: Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 8.4 Index: example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 8.5 Problems with index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 8.6 Inverted Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 8.7 Indexing and Inverted Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 8.8 Index size and space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 8.9 Retrieval Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 9 Query Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 9.1 Query processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 9.2 Query processing: Boolean Model . . . . . . . . . . . . . . . . . . . . . . . . 57 10 Problem with Boolean search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 11 Ranking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 11.1 Ranking: boolean query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 11.2 Full text queries (non boolean) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 11.3 Document and Query as a binary vector . . . . . . . . . . . . . . . . . . . . 62 Contents First Last Prev Next ◭

  5. 11.4 Similarity Measure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 12 Query-document matching scores I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 12.1 Summing up: Weights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 13 Recall: Vector Space Model: Document as vectors . . . . . . . . . . . . . . . . . . 66 13.1 Assumption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 14 Query-document matching scores II . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 14.1 Proximity: Angle and Cosine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 14.2 Example: Length . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 14.3 Example: Cosine Similarity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 14.4 Example: Query-Document Matching . . . . . . . . . . . . . . . . . . . . . 72 15 Summary: Ranked retrieval in the vector space model . . . . . . . . . . . . . . . 73 16 Evaluation Measures: Contingeny Matrix . . . . . . . . . . . . . . . . . . . . . . . . . 74 16.1 Evaluation Setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 16.2 Evaluation Measure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 16.3 Precision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 16.4 Precision, Recall and F-Measure . . . . . . . . . . . . . . . . . . . . . . . . . . 79 16.5 Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 16.6 Problem with Recall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 16.7 Trade-off between Recall & Precision. . . . . . . . . . . . . . . . . . . . . . 83 Contents First Last Prev Next ◭

  6. 16.8 Precision/Recall: at position . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 Contents First Last Prev Next ◭

  7. 1. Today main topics We will see: • HCI: NLIDB, QA, IQA, IR • Evaluation Measures: Accuracy, Precision, Recall. Contents First Last Prev Next ◭

  8. 2. HCI via Natural Language In the ’50 Machine Translation work pointed out serious problems in trying to deal with unrestricted, extended text in open domains. This led researchers in the ’60 and early ’70 to focus on question-answering dialogues in restricted domain. Attention shifted from developing NL systems to solving individual language-related problems, e.g., to develop faster, and more efficient parsers. Now, researchers are back to deal with unrestricted extended text and dialogues. 1. NLIDB: NL interface to DB 2. Dialogue Systems 3. QA: Question Answering 4. IQA: Interactive Question Answering 5. IR: Information Retrieval All of them aim at assisting users to access data from some source. Contents First Last Prev Next ◭

  9. 3. Natural Language Interfaces to Data Bases NLIDB refers to systems that allow the user to access information stored in a database by typing requests in some natural language. Its history (see Androutsopoulos for more details): ’60/’70 they were built having a particular DB in mind. No interest in portability issues. E.g., LUNAR late ’70 Dialogues; large DB; semantic grammars (domain dependent - no portable). E.g. LADDER early ’80 From English into Prolog evaluated against Prolog DB. Eg., CHAT-80 mid ’80 popular research area. Research focused on portability issues. E.g. TEAM ’90 NLIDBs did not gain the expected commercial acceptance. Alternative solutions were successful (graphical or form-based interface). Decrease in the nubmer of papers on the topic. Contents First Last Prev Next ◭

Recommend

Computational Linguistics: Evaluation Methods Raffaella Bernardi University of Trento Contents

Computational Linguistics: Evaluation Methods Raffaella Bernardi University of Trento Contents

Computational Linguistics: Evaluation Methods Raffaella Bernardi University of Trento Contents First Last Prev Next 1. Admin Perusall sends email reminders to students 3, 2, and 1 day before the deadline of an assignment. Only

477 views • 34 slides
Topics in Computational Linguistics Topics in Computational Linguistics March 28, 2014 GIL,

Topics in Computational Linguistics Topics in Computational Linguistics March 28, 2014 GIL,

. . . . .. . . .. . . .. . . .. . . .. .. . . Topics in Computational Linguistics Topics in Computational Linguistics March 28, 2014 GIL, National Taiwan University Lab of Ontologies, Language Processing and e-Humanities

783 views • 56 slides
Foundations of Computational Linguistics man-machine communication in natural language R OLAND H

Foundations of Computational Linguistics man-machine communication in natural language R OLAND H

Foundations of Computational Linguistics man-machine communication in natural language R OLAND H AUSSER Computational Linguistics Universitt Erlangen Nrnberg Germany Foundations of Computational Linguistics viii Part IV Semantics and

1.53k views • 130 slides
Foundations of Computational Linguistics man-machine communication in natural language R OLAND H

Foundations of Computational Linguistics man-machine communication in natural language R OLAND H

Foundations of Computational Linguistics man-machine communication in natural language R OLAND H AUSSER Computational Linguistics Universitt Erlangen Nrnberg Germany Foundations of Computational Linguistics iv Part II Theory of Grammar 7.

1.29k views • 107 slides
Foundations of Computational Linguistics man-machine communication in natural language R OLAND H

Foundations of Computational Linguistics man-machine communication in natural language R OLAND H

Foundations of Computational Linguistics man-machine communication in natural language R OLAND H AUSSER Computational Linguistics Universitt Erlangen Nrnberg Germany Foundations of Computational Linguistics vi Part III Morphology and

1.64k views • 141 slides
Computational linguistics and NLP: How far from generic linguistics? Andrey Kutuzov University

Computational linguistics and NLP: How far from generic linguistics? Andrey Kutuzov University

Computational linguistics and NLP: How far from generic linguistics? Andrey Kutuzov University of Oslo Language Technology Group with thanks to Joachim Nivre and Abigail See January 17, 2018 Contents What is NLP? 1 Case 1: Redefining parts

2.56k views • 189 slides
Computational Linguistics I CMSC 723 / LING 723 / INST 725 M ARINE C ARPUAT marine@cs.umd.edu

Computational Linguistics I CMSC 723 / LING 723 / INST 725 M ARINE C ARPUAT marine@cs.umd.edu

Computational Linguistics I CMSC 723 / LING 723 / INST 725 M ARINE C ARPUAT marine@cs.umd.edu What is Computational Linguistics? Study of computer processing of natural languages Interdisciplinary field Roots in linguistics and

310 views • 27 slides
Computational Linguistics I CMSC 723 / LING 723 / INST 725 M ARINE C ARPUAT marine@cs.umd.edu

Computational Linguistics I CMSC 723 / LING 723 / INST 725 M ARINE C ARPUAT marine@cs.umd.edu

Computational Linguistics I CMSC 723 / LING 723 / INST 725 M ARINE C ARPUAT marine@cs.umd.edu What is Computational Linguistics? Study of computer processing of natural languages Interdisciplinary field Roots in linguistics and

749 views • 28 slides
Foundations of Computational Linguistics man-machine communication in natural language R OLAND H

Foundations of Computational Linguistics man-machine communication in natural language R OLAND H

Foundations of Computational Linguistics man-machine communication in natural language R OLAND H AUSSER Computational Linguistics Universitt Erlangen Nrnberg Germany Foundations of Computational Linguistics ii Part I Theory of Language 1.

957 views • 92 slides
Evaluation measures in NLP Zdenk abokrtsk October 30, 2020 NPFL070 Language Data Resources

Evaluation measures in NLP Zdenk abokrtsk October 30, 2020 NPFL070 Language Data Resources

Evaluation measures in NLP Zdenk abokrtsk October 30, 2020 NPFL070 Language Data Resources Charles University Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics unless otherwise stated Outline Evaluation

470 views • 45 slides
Convergences: Convergences Bitext + morph = IGT concern with data bilingual text

Convergences: Convergences Bitext + morph = IGT concern with data bilingual text

Fieldwork as a Computational Problem three data types The Human Language Project: Uniting Computational Linguistics three kinds of metadata with Documentary Linguistics relations Steven Bird computational challenge

398 views • 10 slides
One-Shot Learning: Language Acquisition for Machine SS16 Computational Linguistics for

One-Shot Learning: Language Acquisition for Machine SS16 Computational Linguistics for

One-Shot Learning: Language Acquisition for Machine SS16 Computational Linguistics for Low-Resource Languages Mayumi Ohta July 6, 2016 Institute for Computational Linguistics Heidelberg University Table of contents 1. Introduction 2.

985 views • 70 slides
Computational Linguistics Statistical NLP Aurlie Herbelot 2020 Centre for Mind/Brain Sciences

Computational Linguistics Statistical NLP Aurlie Herbelot 2020 Centre for Mind/Brain Sciences

Computational Linguistics Statistical NLP Aurlie Herbelot 2020 Centre for Mind/Brain Sciences University of Trento 1 Table of Contents 1. Probabilities and language modeling 2. Naive Bayes algorithm 3. Evaluation issues 4. The feature

973 views • 78 slides
Evaluation measures in NLP Zdenk abokrtsk 8th April 2020 NPFL124 Natural Language

Evaluation measures in NLP Zdenk abokrtsk 8th April 2020 NPFL124 Natural Language

Evaluation measures in NLP Zdenk abokrtsk 8th April 2020 NPFL124 Natural Language Processing Charles University Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics unless otherwise stated Title of your

597 views • 32 slides
Why does NLP need linguistics? Julia Hockenmaier juliahmr@illinois.edu NLP and Linguistics:

Why does NLP need linguistics? Julia Hockenmaier juliahmr@illinois.edu NLP and Linguistics:

Why does NLP need linguistics? Julia Hockenmaier juliahmr@illinois.edu NLP and Linguistics: Finding the Common Ground 48th ACL, July 2010 Uppsala, Sweden 1 Why do computational linguists need to know about linguistics? 2 Why do

838 views • 44 slides
Penalty Functions for Evaluation Measures of Unsegmented Speech Retrieval Petra Galukov,

Penalty Functions for Evaluation Measures of Unsegmented Speech Retrieval Petra Galukov,

Penalty Functions for Evaluation Measures of Unsegmented Speech Retrieval Petra Galukov, Pavel Pecina, Jan Haji Institute of Formal and Applied Linguistics Charles University in Prague {galuscakova,pecina,hajic}@ufal.mff.cuni.cz

520 views • 20 slides
Computational Linguistics II: Parsing Introduction Frank Richter & Jan-Philipp S ohn

Computational Linguistics II: Parsing Introduction Frank Richter & Jan-Philipp S ohn

Computational Linguistics II: Parsing Introduction Frank Richter & Jan-Philipp S ohn fr@sfs.uni-tuebingen.de, jp.soehn@uni-tuebingen.de Computational Linguistics II: Parsing p.1 Oral Presentations The Preparation Phase

589 views • 21 slides
Corpus Linguistics Statistical Measures in Information Retrieval Niko Schenk Institut f ur

Corpus Linguistics Statistical Measures in Information Retrieval Niko Schenk Institut f ur

Introduction N -Gram Measures Homework Corpus Linguistics Statistical Measures in Information Retrieval Niko Schenk Institut f ur England- und Amerikastudien Goethe-Universit at Frankfurt am Main Winter Term 2015/2016 January 10, 2017

766 views • 60 slides
Computational Linguistics: TAG, CG and DG Raffaella Bernardi University of Trento Contents

Computational Linguistics: TAG, CG and DG Raffaella Bernardi University of Trento Contents

Computational Linguistics: TAG, CG and DG Raffaella Bernardi University of Trento Contents First Last Prev Next 1. Last time and today We have seen that Formal Grammars play a crucial role in the research on Computational

1.04k views • 40 slides
CMPT 413 Computational Linguistics Anoop Sarkar http://www.cs.sfu.ca/~anoop Finite-state

CMPT 413 Computational Linguistics Anoop Sarkar http://www.cs.sfu.ca/~anoop Finite-state

CMPT 413 Computational Linguistics Anoop Sarkar http://www.cs.sfu.ca/~anoop Finite-state transducers Many applications in Other applications computational include: linguistics Grapheme to phoneme Text normalization

409 views • 36 slides
Computational Linguistics II: Parsing Summing up CF Languages: Derivations Frank Richter &

Computational Linguistics II: Parsing Summing up CF Languages: Derivations Frank Richter &

Computational Linguistics II: Parsing Summing up CF Languages: Derivations Frank Richter & Jan-Philipp S ohn fr@sfs.uni-tuebingen.de, jp.soehn@uni-tuebingen.de Computational Linguistics II: Parsing p.1 The Big Picture hierarchy

358 views • 7 slides
Computational Linguistics II: Parsing CSGs, Turing Machines, LBAs Outlook: Other Grammar

Computational Linguistics II: Parsing CSGs, Turing Machines, LBAs Outlook: Other Grammar

Computational Linguistics II: Parsing CSGs, Turing Machines, LBAs Outlook: Other Grammar Formalisms Frank Richter & Jan-Philipp S ohn fr@sfs.uni-tuebingen.de, jp.soehn@uni-tuebingen.de Computational Linguistics II: Parsing p.1 The

453 views • 28 slides
9. Evaluation Outline 9.1. Cranfield Paradigm & TREC 9.2. Non-Traditional Measures 9.3.

9. Evaluation Outline 9.1. Cranfield Paradigm & TREC 9.2. Non-Traditional Measures 9.3.

9. Evaluation Outline 9.1. Cranfield Paradigm & TREC 9.2. Non-Traditional Measures 9.3. Incomplete Judgments 9.4. Low-Cost Evaluation 9.5. Crowdsourcing 9.6. Online Evaluation Advanced Topics in Information Retrieval / Evaluation 2

573 views • 40 slides
Computational Linguistics The Future of NLP We can study anything about language ... 1.

Computational Linguistics The Future of NLP We can study anything about language ... 1.

Computational Linguistics The Future of NLP We can study anything about language ... 1. Formalize some insights 2. Study the formalism mathematically A Few Random Remarks 3. Develop & implement algorithms 4. Test on real

362 views • 4 slides

More recommend