evaluation metrics for machine
play

Evaluation Metrics for Machine Reading Comprehension (RC): - PowerPoint PPT Presentation

Jan 04, 2023 •199 likes •577 views

Evaluation Metrics for Machine Reading Comprehension (RC): Prerequisite Skills and Readability Sugawara et al. The University of Tokyo, Fujitsu Laboratories Ltd., Natural Institute of Informatics Presented by: Shaima AbdulMajeed To give the

  1. Evaluation Metrics for Machine Reading Comprehension (RC): Prerequisite Skills and Readability Sugawara et al. The University of Tokyo, Fujitsu Laboratories Ltd., Natural Institute of Informatics Presented by: Shaima AbdulMajeed

  2. To give the agent the ability to: RC 1. Read open-domain documents Task 2. Answer questions about them

  3. Knowing Quality of Reading Goal Comprehension (RC) datasets

  4. To know Which dataset to use that Why best evaluates the developed RC system

  5. RC dataset Example

  6. Datasets evaluated

  7. Current dataset metrics • Question types • Answer types • Categories

  8. Is that enough ?

  9. Does readability of text correlates with difficulty of answering questions about it?

  10. Evaluation Metrics Proposed 1. Prerequisite skills 2. Readability metrics

  11. 1. Object Tracking 2. Mathematical Reasoning 3. Coreference resolution 4. Logical Reasoning 5. Analogy 6. Causal relation Prerequisite skills 7. Spatiotemporal relation 8. Ellipsis 9. Bridging 10. Elaboration 11. Meta-Knowledge 12. Schematic clause relation 13. Punctuation

  12. Tracking or grasping of multiple objects 1. Object Tracking 2. Mathematical Reasoning 3. Coreference resolution 4. Logical Reasoning 5. Analogy Context: Tom ate apples. 6. Causal relation 7. Spatiotemporal relation Mary ate apples, too. 8. Ellipsis 9. Bridging Q: Who ate apples? 10. Elaboration 11. Meta-Knowledge A: Tom and Mary 12. Schematic clause relation 13. Punctuation (Object: Tom, Mary)

  13. Statistical, mathematical and 1. Object Tracking quantitative reasoning 2. Mathematical Reasoning 3. Coreference resolution 4. Logical Reasoning 5. Analogy 6. Causal relation Context: Tom ate ten apples. 7. Spatiotemporal relation Mary ate eight apples. 8. Ellipsis 9. Bridging Q: How many apples did 10. Elaboration 11. Meta-Knowledge Tom and Mary eat? 12. Schematic clause relation 13. Punctuation A: eighteen

  14. Detection and resolution of all 1. Object Tracking possible demonstratives 2. Mathematical Reasoning 3. Coreference resolution 4. Logical Reasoning 5. Analogy Context: Tom was hungry. 6. Causal relation 7. Spatiotemporal relation He ate ten apples. 8. Ellipsis 9. Bridging Q: How many apples did 10. Elaboration 11. Meta-Knowledge Tom eat? 12. Schematic clause relation 13. Punctuation A: ten (Tom = He)

  15. 1. Object Tracking Understanding of Predicate Logic 2. Mathematical Reasoning 3. Coreference resolution 4. Logical Reasoning 5. Analogy Context: All students have a 6. Causal relation 7. Spatiotemporal relation pen. Tom is a student. 8. Ellipsis 9. Bridging Q: Does Tom have a pen. 10. Elaboration 11. Meta-Knowledge A: Yes (and object tracking) 12. Schematic clause relation 13. Punctuation

  16. Understanding metaphors Context: The White House 1. Object Tracking 2. Mathematical Reasoning said Trump is open to … 3. Coreference resolution 4. Logical Reasoning 5. Analogy 6. Causal relation Q: Did the President of the 7. Spatiotemporal relation 8. Ellipsis United States and his staff 9. Bridging 10. Elaboration say Trump is open to ... 11. Meta-Knowledge 12. Schematic clause relation A: Yes 13. Punctuation (The White House said = POTUS and his staff said...)

  17. 1. Object Tracking 2. Mathematical Reasoning 3. Coreference resolution 4. Logical Reasoning 5. Analogy 6. Causal relation “why,” “because,” 7. Spatiotemporal relation 8. Ellipsis 9. Bridging 10. Elaboration 11. Meta-Knowledge 12. Schematic clause relation 13. Punctuation

  18. Context: One day, Tom went to the park. After that, 1. Object Tracking he went to the restaurant. 2. Mathematical Reasoning 3. Coreference resolution Finally, he went to his 4. Logical Reasoning 5. Analogy grandma's house. 6. Causal relation 7. Spatiotemporal relation 8. Ellipsis 9. Bridging Q: Where did Tom go 10. Elaboration 11. Meta-Knowledge finally? 12. Schematic clause relation A: his grandma's house 13. Punctuation (Finally: temporal)

  19. Recognizing implicit 1. Object Tracking 2. Mathematical Reasoning information 3. Coreference resolution 4. Logical Reasoning 5. Analogy 6. Causal relation 7. Spatiotemporal relation She is a smart student 8. Ellipsis 9. Bridging = 10. Elaboration 11. Meta-Knowledge 12. Schematic clause relation She is a student 13. Punctuation

  20. Inference supported by 1. Object Tracking 2. Mathematical Reasoning grammatical and lexical 3. Coreference resolution 4. Logical Reasoning knowledge 5. Analogy 6. Causal relation She loves sushi 7. Spatiotemporal relation 8. Ellipsis 9. Bridging = 10. Elaboration 11. Meta-Knowledge 12. Schematic clause relation She likes sushi 13. Punctuation

  21. Inference using known facts, general knowledge 1. Object Tracking 2. Mathematical Reasoning 3. Coreference resolution 4. Logical Reasoning 5. Analogy The writer of Hamlet 6. Causal relation 7. Spatiotemporal relation was Shakespeare 8. Ellipsis 9. Bridging � 10. Elaboration 11. Meta-Knowledge Shakespeare wrote 12. Schematic clause relation 13. Punctuation Hamlet

  22. 1. Object Tracking Who are the principal 2. Mathematical Reasoning 3. Coreference resolution characters of the story? 4. Logical Reasoning 5. Analogy 6. Causal relation 7. Spatiotemporal relation 8. Ellipsis 9. Bridging What is the main 10. Elaboration 11. Meta-Knowledge subject of this article? 12. Schematic clause relation 13. Punctuation

  23. Understanding of complex 1. Object Tracking sentences that have 2. Mathematical Reasoning 3. Coreference resolution coordination or subordination 4. Logical Reasoning 5. Analogy Context: Tom has a friend 6. Causal relation 7. Spatiotemporal relation whose name is John. 8. Ellipsis 9. Bridging 10. Elaboration 11. Meta-Knowledge Q: What is a name of 12. Schematic clause relation 13. Punctuation Tom's friend? A: John (whose = relative clause)

  24. Understanding of punctuation marks 1. Object Tracking Context: The AFC champion (Denver 2. Mathematical Reasoning Broncos) defeated the NFC champion 3. Coreference resolution 4. Logical Reasoning (Carolina Panthers) in super bowl 50 5. Analogy 6. Causal relation 7. Spatiotemporal relation 8. Ellipsis Q: Which NFL team won Super Bowl 9. Bridging 50? 10. Elaboration 11. Meta-Knowledge A: Denver Broncos 12. Schematic clause relation 13. Punctuation Note: parentheses present the champion team's name

  25. 1. Lexical Features Readability 2. Syntactic Features metrics 3. Traditional Features

  26. Step 1: annotators see simultaneously the context, question, and its answer e.g. Q: Why Tom looked angry? A: His sister ate his cake. Step 2: Select sentences (from the context) e.g. Context: Annotation (C1) Tom is a student. (C2) Tom looks annoyed because his sister ate his cake. Procedure (C3) His sister's name is Sylvia. (100 Qu) -> Select: C2 Step 3: Select skills required for answering the question e.g.: C2: Tom looks annoyed because his sister ate his cake. � Skill: causal relation ("because"), bridging (lexical knowledge of "annoyed = angry")

  27. 1. Prerequisite skills required for each RC dataset 2. Prerequisite skills required per question Results 3. Readability of each RC dataset 4. Correlation between readability and prerequisite skills required.

  28. Results 1- prerequisite skills required for each RC dataset 1. QA4MRE (Highest score in all skills): • Bridging • Elaboration • Clause Relation • Punctuation 2. MCTest • Casual Relation • Meta Knowledge • Coreference resolution • Spatiotemoral Relation

  29. Results 2- Number of prerequisite skills required per question QA4MRE MCTests SQuAD WDW MS News MARCO QA Avg 3.25 1.56 1.28 2.43 1.19 1.99 Highest – technical documents – Qu handcrafted by experts

  30. Results Nonsense/Difficult Questions QA4MRE MCTest SQuAD WDW MARCO News QA Non 10 1 3 27 14 1 sense

  31. Results 3- Readability metrics for each RC dataset QA4MRE MCTests SQuAD WDW MARCO News QA F-K 14.9 3.6 14.6 15.3 12.1 12.6

  32. Results 4- Correlation between readability metrics and the number of required prerequisite skills

  33. Results 4- Correlation between readability metrics and the number of required prerequisite skills

  34. Summary QA4MRE MCTest SQuAD • Hard to read • Easy to read • Hard to read • Hard to answer • Hard to answer • Easy to answer

  35. How to utilize this study 1. Preparing appropriate datasets for each step of RC dev: I. easy-to-read and easy-to-answer II. easy-to-read but difficult-to-answer dataset III. difficult-to-read and difficult-to-answer datasets 2. Apply metrics to evaluate other datasets

Recommend

Chapter 8 Evaluation Statistical Machine Translation Evaluation How good is a given machine

Chapter 8 Evaluation Statistical Machine Translation Evaluation How good is a given machine

Chapter 8 Evaluation Statistical Machine Translation Evaluation How good is a given machine translation system? Hard problem, since many different translations acceptable semantic equivalence / similarity Evaluation metrics

456 views • 42 slides
Automated Metrics for MT Evaluation 11731: 11731: Machine Translation Alon Lavie

Automated Metrics for MT Evaluation 11731: 11731: Machine Translation Alon Lavie

Automated Metrics for MT Evaluation 11731: 11731: Machine Translation Alon Lavie February 14, 2013 Automated Metrics for MT Evaluation Idea: compare output of an MT system to a reference good (usually human) translation:

1.07k views • 50 slides
CORE METRICS & UNIVERSITY BENCHMARKS Core Metric Subject or Programme University Benchmark

CORE METRICS & UNIVERSITY BENCHMARKS Core Metric Subject or Programme University Benchmark

ANNUAL MONITORING 2020 (ABOUT 2019-20) Inputs: Data analysis: Core Metrics, University KPIs, Gap Metrics (new) , External Examiners Reports Benchmarks set by Office for Students for University Evaluation: Evaluation of performance

560 views • 12 slides
A New Method for the Study of Correlations between MT Evaluation Metrics Paula Estrella Andrei

A New Method for the Study of Correlations between MT Evaluation Metrics Paula Estrella Andrei

A New Method for the Study of Correlations between MT Evaluation Metrics Paula Estrella Andrei Popescu-Belis Margaret King School of Translation and Interpreting University of Geneva Introduction Correlation with human metrics is a

541 views • 24 slides
Model Evaluation Model Evaluation Metrics for Performance Evaluation How to evaluate the

Model Evaluation Model Evaluation Metrics for Performance Evaluation How to evaluate the

Model Evaluation Model Evaluation Metrics for Performance Evaluation How to evaluate the performance of a model? Methods for Performance Evaluation How to obtain reliable estimates? Methods for Model Comparison How to

636 views • 36 slides
Machine Spaces: Axioms and Metrics J org Zimmermann and Armin B. Cremers Institute of Computer

Machine Spaces: Axioms and Metrics J org Zimmermann and Armin B. Cremers Institute of Computer

Machine Spaces: Axioms and Metrics J org Zimmermann and Armin B. Cremers Institute of Computer Science University of Bonn, Germany 1 J org Zimmermann and Armin B. Cremers: Machine Spaces: Axioms and Metrics Machine Spaces: Motivation and

478 views • 19 slides
What we learned from Community Metrics Agenda Why are metrics used? How metrics are used

What we learned from Community Metrics Agenda Why are metrics used? How metrics are used

What we learned from Community Metrics Agenda Why are metrics used? How metrics are used in two Open Source communities Common pitfalls and ways to avoid them Why use Metrics? Transparency Who is contributing to the

166 views • 14 slides
Measuring Non-Expert Comprehension of Machine Learning Fairness Metrics Debjani Saha , Candice

Measuring Non-Expert Comprehension of Machine Learning Fairness Metrics Debjani Saha , Candice

Measuring Non-Expert Comprehension of Machine Learning Fairness Metrics Debjani Saha , Candice Schumann, Duncan C. McElfresh, John P. Dickerson, Michelle L. Mazurek, Michael Carl Tschantz 37th International Conference on Machine Learning (ICML)

980 views • 69 slides
Evaluation of example tools for hairy tasks. Presenter: Hardik Sahi (20743327) Outline

Evaluation of example tools for hairy tasks. Presenter: Hardik Sahi (20743327) Outline

Evaluation of example tools for hairy tasks. Presenter: Hardik Sahi (20743327) Outline Definition of a hairy task. Metrics for tool evaluation When is recall favoured over precision? New metrics: Weighted F measure,

244 views • 21 slides
History & Evaluation CMSC 470 Marine Carpuat T odays topics Machine Translation

History & Evaluation CMSC 470 Marine Carpuat T odays topics Machine Translation

Machine Translation History & Evaluation CMSC 470 Marine Carpuat T odays topics Machine Translation Context: Historical Background Machine Translation is an old idea Machine Translation Evaluation 1947 When I look at an

825 views • 40 slides
Machine Translation History & Evaluation CMSC 470 Marine Carpuat T odays topics

Machine Translation History & Evaluation CMSC 470 Marine Carpuat T odays topics

Machine Translation History & Evaluation CMSC 470 Marine Carpuat T odays topics Machine Translation Context: Historical Background Machine Translation Evaluation 1947 When I look at an article in Russian, I say to myself:

505 views • 38 slides
Repeated On-Chip Interconnect Analysis and Evaluation of Delay, Power, and Bandwidth Metrics under

Repeated On-Chip Interconnect Analysis and Evaluation of Delay, Power, and Bandwidth Metrics under

Repeated On-Chip Interconnect Analysis and Evaluation of Delay, Power, and Bandwidth Metrics under Different Design Goals Ling Zhang 1 , Hongyu Chen 2 , Bo Yao 3 , Kevin Hamilton 4 , Chung-Kuan Cheng 1 1 University of California, San Diego, CA

193 views • 6 slides
Benchmarking and Evaluation in Inverse Reinforcement Learning Peter Henderson Workshop on New

Benchmarking and Evaluation in Inverse Reinforcement Learning Peter Henderson Workshop on New

Benchmarking and Evaluation in Inverse Reinforcement Learning Peter Henderson Workshop on New Benchmarks, Metrics, and Competitions for Robotic Learning RSS 2018 Where do you see the shortcomings in existing benchmarks and evaluation metrics ?

722 views • 24 slides
Evaluation Experimental protocols, datasets, metrics Web Search 1 What makes a good search

Evaluation Experimental protocols, datasets, metrics Web Search 1 What makes a good search

Evaluation Experimental protocols, datasets, metrics Web Search 1 What makes a good search engine? Efficiency : It replies to user queries without noticeable delays. 1 sec is the limit for users feeling that they are freely

650 views • 35 slides
NYS Ending the Epidemic Metrics Where we were, where we are, and where we need to be by the end

NYS Ending the Epidemic Metrics Where we were, where we are, and where we need to be by the end

NYS Ending the Epidemic Metrics Where we were, where we are, and where we need to be by the end of 2020 James Tesoriero, Ph.D Division of HIV/STI Epidemiology, Evaluation and Partner Services NYSDOH AIDS Institute 2 Objectives Briefly

445 views • 26 slides
Evaluation Philipp Koehn 22 September 2020 Philipp Koehn Machine Translation: Evaluation 22

Evaluation Philipp Koehn 22 September 2020 Philipp Koehn Machine Translation: Evaluation 22

Evaluation Philipp Koehn 22 September 2020 Philipp Koehn Machine Translation: Evaluation 22 September 2020 Evaluation 1 How good is a given machine translation system? Hard problem, since many different translations acceptable

787 views • 50 slides
ICS 667 Advanced HCI Design Methods 08. Intro to Evaluation Analytic Evaluation Dan Suthers

ICS 667 Advanced HCI Design Methods 08. Intro to Evaluation Analytic Evaluation Dan Suthers

ICS 667 Advanced HCI Design Methods 08. Intro to Evaluation Analytic Evaluation Dan Suthers Spring 2005 Outline Introduction to Evaluation Types of Evaluation etc. Usability Specifications Analytic Methods Metrics and

519 views • 21 slides
AB3DMOT: A Baseline for 3D Multi-Object Tracking and New Evaluation Metrics Xinshuo Weng, Jianren

AB3DMOT: A Baseline for 3D Multi-Object Tracking and New Evaluation Metrics Xinshuo Weng, Jianren

AB3DMOT: A Baseline for 3D Multi-Object Tracking and New Evaluation Metrics Xinshuo Weng, Jianren Wang, David Held, Kris Kitani Robotics Institute, Carnegie Mellon University European Conference on Computer Vision (ECCV) Workshops , 2020 1

642 views • 21 slides
AGENCY OPERATIONS METRICS The Metrics of Me The Metrics of Me x 159 13,006 5 days old books

AGENCY OPERATIONS METRICS The Metrics of Me The Metrics of Me x 159 13,006 5 days old books

Dont trust your gut AGENCY OPERATIONS METRICS The Metrics of Me The Metrics of Me x 159 13,006 5 days old books read so far in 2018 The Metrics of Me 6.3 4.5 4 0 hours of sleep hours in dance class hours cooking hours commuting

549 views • 20 slides
BBM406 Fundamentals of Machine Learning Lecture 3: Kernel Regression, Distance Metrics, Curse

BBM406 Fundamentals of Machine Learning Lecture 3: Kernel Regression, Distance Metrics, Curse

photo:@rewardyfahmi // Unsplash BBM406 Fundamentals of Machine Learning Lecture 3: Kernel Regression, Distance Metrics, Curse of Dimensionality Aykut Erdem // Hacettepe University // Fall 2019 Administrative Assignment 1 will be out

1.25k views • 32 slides
The Curse of Class Imbalance and Conflicting Metrics with Machine Learning for Side-channel

The Curse of Class Imbalance and Conflicting Metrics with Machine Learning for Side-channel

The Curse of Class Imbalance and Conflicting Metrics with Machine Learning for Side-channel Evaluations Stjepan Picek, Annelie Heuser, Alan Jovic, Shivam Bhasin, and Francesco Regazzoni Big Picture side-channel measurements device classifier

822 views • 33 slides
3D Multi-Object Tracking: A Baseline and New Evaluation Metrics Xinshuo Weng, Jianren Wang, David

3D Multi-Object Tracking: A Baseline and New Evaluation Metrics Xinshuo Weng, Jianren Wang, David

3D Multi-Object Tracking: A Baseline and New Evaluation Metrics Xinshuo Weng, Jianren Wang, David Held, Kris Kitani Robotics Institute, Carnegie Mellon University IEEE/RSJ International Conference on Intelligent Robots and Systems ( IROS ) , 2020

659 views • 29 slides
Evaluation Metrics Presented by Dawn Lawrie 1 Some Possibilities Precision Recall F-measure

Evaluation Metrics Presented by Dawn Lawrie 1 Some Possibilities Precision Recall F-measure

Evaluation Metrics Presented by Dawn Lawrie 1 Some Possibilities Precision Recall F-measure Mean Average Precision Mean Reciprocal Rank 2 Precision Set Proportion of things of interest in some set Example: Im interested in apples

555 views • 28 slides
Metrics and models for Web performance evaluation or, How to measure SpeedIndex from raw encrypted

Metrics and models for Web performance evaluation or, How to measure SpeedIndex from raw encrypted

Metrics and models for Web performance evaluation or, How to measure SpeedIndex from raw encrypted packets, and why it matters QoE = f(QoS) This talk Dario Rossi dario.rossi@huawei.com Director, DataCom (*) Lab Brussels, Feb 1 st 2020 Huawei

557 views • 29 slides

More recommend