the impact of distributional metrics in the quality of
play

The Impact of Distributional Metrics in the Quality of Relational - PowerPoint PPT Presentation

Apr 10, 2023 •45 likes •665 views

The Impact of Distributional Metrics in the Quality of Relational Triples calo Oliveira 1 , Paulo Gomes Hernani Costa, Hugo Gon hpcosta@student.dei.uc.pt, { hroliv,pgomes } @dei.uc.pt Cognitive & Media Systems Group CISUC, University of

  1. The Impact of Distributional Metrics in the Quality of Relational Triples calo Oliveira 1 , Paulo Gomes Hernani Costa, Hugo Gon¸ hpcosta@student.dei.uc.pt, { hroliv,pgomes } @dei.uc.pt Cognitive & Media Systems Group CISUC, University of Coimbra Lisbon, August 16, 2010 1supported by FCT scholarship grant SFRH/BD/44955/2008 Costa, Gon¸ calo Oliveira & Gomes (CISUC) LaTeCH 2010 Lisbon, August 16, 2010 1 / 21

  2. Outline Introduction 1 Information Extraction Information Retrieval Research Goals Approach 2 Experimentation 3 Set-up Metrics adaptation Results Additional experimentation Concluding remarks 4 Costa, Gon¸ calo Oliveira & Gomes (CISUC) LaTeCH 2010 Lisbon, August 16, 2010 2 / 21

  3. Introduction Introduction Knowledge bases (eg. WordNet) are useful resources for NLP Costa, Gon¸ calo Oliveira & Gomes (CISUC) LaTeCH 2010 Lisbon, August 16, 2010 3 / 21

  4. Introduction Introduction Knowledge bases (eg. WordNet) are useful resources for NLP Their creation and maintenance involves intensive human effort Costa, Gon¸ calo Oliveira & Gomes (CISUC) LaTeCH 2010 Lisbon, August 16, 2010 3 / 21

  5. Introduction Introduction Knowledge bases (eg. WordNet) are useful resources for NLP Their creation and maintenance involves intensive human effort Automatic creation/enrichment from textual resources is an alternative Costa, Gon¸ calo Oliveira & Gomes (CISUC) LaTeCH 2010 Lisbon, August 16, 2010 3 / 21

  6. Introduction Introduction Knowledge bases (eg. WordNet) are useful resources for NLP Their creation and maintenance involves intensive human effort Automatic creation/enrichment from textual resources is an alternative ▶ Higher coverage, easier update, but... Costa, Gon¸ calo Oliveira & Gomes (CISUC) LaTeCH 2010 Lisbon, August 16, 2010 3 / 21

  7. Introduction Introduction Knowledge bases (eg. WordNet) are useful resources for NLP Their creation and maintenance involves intensive human effort Automatic creation/enrichment from textual resources is an alternative ▶ Higher coverage, easier update, but... ▶ Precision is lower Costa, Gon¸ calo Oliveira & Gomes (CISUC) LaTeCH 2010 Lisbon, August 16, 2010 3 / 21

  8. Introduction Introduction Knowledge bases (eg. WordNet) are useful resources for NLP Their creation and maintenance involves intensive human effort Automatic creation/enrichment from textual resources is an alternative ▶ Higher coverage, easier update, but... ▶ Precision is lower ▶ Evaluation requires once again intensive human labour! Costa, Gon¸ calo Oliveira & Gomes (CISUC) LaTeCH 2010 Lisbon, August 16, 2010 3 / 21

  9. Introduction Information Extraction Information extraction (IE) Automatic extraction of structured information from natural language. Costa, Gon¸ calo Oliveira & Gomes (CISUC) LaTeCH 2010 Lisbon, August 16, 2010 4 / 21

  10. Introduction Information Extraction Information extraction (IE) Automatic extraction of structured information from natural language. “Car is a vehicle with 4 wheels and an engine, used for carrying a small number of passengers.” Costa, Gon¸ calo Oliveira & Gomes (CISUC) LaTeCH 2010 Lisbon, August 16, 2010 4 / 21

  11. Introduction Information Extraction Information extraction (IE) Automatic extraction of structured information from natural language. “Car is a vehicle with 4 wheels and an engine, used for carrying a small number of passengers.” ▶ vehicle HYPERNYM OF car Costa, Gon¸ calo Oliveira & Gomes (CISUC) LaTeCH 2010 Lisbon, August 16, 2010 4 / 21

  12. Introduction Information Extraction Information extraction (IE) Automatic extraction of structured information from natural language. “Car is a vehicle with 4 wheels and an engine, used for carrying a small number of passengers.” ▶ vehicle HYPERNYM OF car ▶ wheel PART OF car ▶ engine PART OF car Costa, Gon¸ calo Oliveira & Gomes (CISUC) LaTeCH 2010 Lisbon, August 16, 2010 4 / 21

  13. Introduction Information Extraction Information extraction (IE) Automatic extraction of structured information from natural language. “Car is a vehicle with 4 wheels and an engine, used for carrying a small number of passengers.” ▶ vehicle HYPERNYM OF car ▶ wheel PART OF car ▶ engine PART OF car ▶ carrying people PURPOSE OF car Costa, Gon¸ calo Oliveira & Gomes (CISUC) LaTeCH 2010 Lisbon, August 16, 2010 4 / 21

  14. Introduction Information Retrieval Information retrieval (IR) Locating specific information in natural language resouces. Costa, Gon¸ calo Oliveira & Gomes (CISUC) LaTeCH 2010 Lisbon, August 16, 2010 5 / 21

  15. Introduction Information Retrieval Information retrieval (IR) Locating specific information in natural language resouces. Approaches based on the occurrence of words in documents. Costa, Gon¸ calo Oliveira & Gomes (CISUC) LaTeCH 2010 Lisbon, August 16, 2010 5 / 21

  16. Introduction Information Retrieval Information retrieval (IR) Locating specific information in natural language resouces. Approaches based on the occurrence of words in documents. Distributional similarity metrics ▶ Cocitation (Small (1973)) ▶ LSA (Deerwester et al. (1990)) ▶ Lin’s (Lin (1998)) ▶ PMI-IR (Turney (2001)) ▶ 휎 (Kozima and Furugori (1993)) ▶ ... Costa, Gon¸ calo Oliveira & Gomes (CISUC) LaTeCH 2010 Lisbon, August 16, 2010 5 / 21

  17. Introduction Research Goals Goals 1 Use IR metrics to improve IE precision ▶ Adapt distributional metrics to determine words similarity Costa, Gon¸ calo Oliveira & Gomes (CISUC) LaTeCH 2010 Lisbon, August 16, 2010 6 / 21

  18. Introduction Research Goals Goals 1 Use IR metrics to improve IE precision ▶ Adapt distributional metrics to determine words similarity ▶ Wandmacher et al. (2007) and Cederberg and Widdows (2003) used LSA to weight hypernymy triples Costa, Gon¸ calo Oliveira & Gomes (CISUC) LaTeCH 2010 Lisbon, August 16, 2010 6 / 21

  19. Introduction Research Goals Goals 1 Use IR metrics to improve IE precision ▶ Adapt distributional metrics to determine words similarity ▶ Wandmacher et al. (2007) and Cederberg and Widdows (2003) used LSA to weight hypernymy triples ▶ What about other semantic relations? Costa, Gon¸ calo Oliveira & Gomes (CISUC) LaTeCH 2010 Lisbon, August 16, 2010 6 / 21

  20. Introduction Research Goals Goals 1 Use IR metrics to improve IE precision ▶ Adapt distributional metrics to determine words similarity ▶ Wandmacher et al. (2007) and Cederberg and Widdows (2003) used LSA to weight hypernymy triples ▶ What about other semantic relations? ▶ What metrics should be used? Costa, Gon¸ calo Oliveira & Gomes (CISUC) LaTeCH 2010 Lisbon, August 16, 2010 6 / 21

  21. Introduction Research Goals Goals 1 Use IR metrics to improve IE precision ▶ Adapt distributional metrics to determine words similarity ▶ Wandmacher et al. (2007) and Cederberg and Widdows (2003) used LSA to weight hypernymy triples ▶ What about other semantic relations? ▶ What metrics should be used? ▶ New combined metrics? Costa, Gon¸ calo Oliveira & Gomes (CISUC) LaTeCH 2010 Lisbon, August 16, 2010 6 / 21

  22. Introduction Research Goals Goals 1 Use IR metrics to improve IE precision ▶ Adapt distributional metrics to determine words similarity ▶ Wandmacher et al. (2007) and Cederberg and Widdows (2003) used LSA to weight hypernymy triples ▶ What about other semantic relations? ▶ What metrics should be used? ▶ New combined metrics? 2 Help manual evaluation Costa, Gon¸ calo Oliveira & Gomes (CISUC) LaTeCH 2010 Lisbon, August 16, 2010 6 / 21

  23. Approach IE system Grammars Extraction of Corpus relational triples Removal of triples with stopwords Lemmatisation Additional Metrics extraction of triples application Costa, Gon¸ calo Oliveira & Gomes (CISUC) LaTeCH 2010 Lisbon, August 16, 2010 7 / 21

  24. Experimentation Set-up Experimentation set-up ublico 2 corpus (annotated version) CETEMP´ ▶ 28,000 documents ▶ 30,100 unique context words (nouns, verbs and adjectives) ▶ term-document matrix 2 http://www.linguateca.pt/cetempublico/ Costa, Gon¸ calo Oliveira & Gomes (CISUC) LaTeCH 2010 Lisbon, August 16, 2010 8 / 21

  25. Experimentation Set-up Experimentation set-up ublico 2 corpus (annotated version) CETEMP´ ▶ 28,000 documents ▶ 30,100 unique context words (nouns, verbs and adjectives) ▶ term-document matrix Triples obtained ▶ Extracted: 20,308 ▶ Discarded: 5,844 ▶ Inferred: 2,492 ▶ Final triple set: 16,956 2 http://www.linguateca.pt/cetempublico/ Costa, Gon¸ calo Oliveira & Gomes (CISUC) LaTeCH 2010 Lisbon, August 16, 2010 8 / 21

  26. Experimentation Metrics adaptation Similarity between two documents For instance, Cocitation: First presented as a similarity metric between scientific papers (Small (1973)) Cocitation ( d i , d j ) = P ( d i ∩ d j ) (1) P ( d i ∪ d j ) Costa, Gon¸ calo Oliveira & Gomes (CISUC) LaTeCH 2010 Lisbon, August 16, 2010 9 / 21

  27. Experimentation Metrics adaptation Similarity between two documents For instance, Cocitation: First presented as a similarity metric between scientific papers (Small (1973)) Cocitation ( d i , d j ) = P ( d i ∩ d j ) (1) P ( d i ∪ d j ) ▶ d i , d j represent two documents Costa, Gon¸ calo Oliveira & Gomes (CISUC) LaTeCH 2010 Lisbon, August 16, 2010 9 / 21

Recommend

THE DISTRIBUTIONAL IMPACT OF POPULATION AGEING Affording Our Future Conference 10-11 December,

THE DISTRIBUTIONAL IMPACT OF POPULATION AGEING Affording Our Future Conference 10-11 December,

THE DISTRIBUTIONAL IMPACT OF POPULATION AGEING Affording Our Future Conference 10-11 December, 2012 Omar A. Aziz, Chris Ball, John Creedy and Jesse Unger LIVING STANDARDS: THE HEART OF OUR POLICY ADVICE Assess the impact of policy across key

509 views • 28 slides
Software Quality Metrics What does software quality mean? and How is it measured? Factors in

Software Quality Metrics What does software quality mean? and How is it measured? Factors in

Software Quality Metrics What does software quality mean? and How is it measured? Factors in Software Quality Factor Criteria Metric Maintainability Simplicity Modularity Conciseness Nesting Level Modules that Percentage of exceed size

256 views • 8 slides
PANEL 4: METRICS OF PROGRESS AND MEASURES OF IMPACT INCLUDING COST- EFFECTIVENESS, OUTCOMES TO

PANEL 4: METRICS OF PROGRESS AND MEASURES OF IMPACT INCLUDING COST- EFFECTIVENESS, OUTCOMES TO

PANEL 4: METRICS OF PROGRESS AND MEASURES OF IMPACT INCLUDING COST- EFFECTIVENESS, OUTCOMES TO VALUE TO PAYERS, INFLUENCING QUALITY OF CARE THROUGH LEARNING HEALTHCARE SYSTEMS Genomic Medicine VIII This meeting will help NHGRI and its Genomic

542 views • 28 slides
The Distributional Impact of Tax-Benefit Systems in Six African Countries Katrin Gasior, Chrysa

The Distributional Impact of Tax-Benefit Systems in Six African Countries Katrin Gasior, Chrysa

The Distributional Impact of Tax-Benefit Systems in Six African Countries Katrin Gasior, Chrysa Leventi, Michael Noble, Gemma Wright & Helen Barnes WIDER Development Conference, Helsinki, 13 th September 2018 Background Taxation and

356 views • 17 slides
Project Management Metrics SE 350 Software Processes & Product Quality Project Management

Project Management Metrics SE 350 Software Processes & Product Quality Project Management

Project Management Metrics SE 350 Software Processes & Product Quality Project Management Metrics Cycletime Productivity Staffing Requirements volatility Reuse metrics Activity progress measurement Estimation

325 views • 21 slides
Medical Device Quality Metrics FDA/Xavier University Initiative MDIC Case for Quality Forum June

Medical Device Quality Metrics FDA/Xavier University Initiative MDIC Case for Quality Forum June

Medical Device Quality Metrics FDA/Xavier University Initiative MDIC Case for Quality Forum June 28, 2016 Purpose and Outcome Purpose: To provide a system of metrics that Supports the Case for Quality Spans the Total Product Lifecycle

635 views • 31 slides
Fundamentals SE 350 Software Process & Product Quality Lecture Objectives Provide some

Fundamentals SE 350 Software Process & Product Quality Lecture Objectives Provide some

Measurement and Metrics Fundamentals SE 350 Software Process & Product Quality Lecture Objectives Provide some basic concepts of metrics Quality attribute metrics and measurements Reliability, validity, error Correlation

896 views • 37 slides
Distributional Semantics The unsupervised modeling of meaning on a large scale Tim Van de Cruys

Distributional Semantics The unsupervised modeling of meaning on a large scale Tim Van de Cruys

equipe melodi Distributional Semantics The unsupervised modeling of meaning on a large scale Tim Van de Cruys IRIT, Toulouse Tuesday 17 November 2015 Distributional similarity distributional hypothesis [Harris 1954] 2/52 Distributional

1.01k views • 88 slides
Quality Metrics Work Group November 16, 2015 1 Agenda Introductions SIM and Care

Quality Metrics Work Group November 16, 2015 1 Agenda Introductions SIM and Care

SIM Quality Metrics Work Group November 16, 2015 1 Agenda Introductions SIM and Care Coordination Overview Overview of Quality Metrics Work Group Discussion on Quality Performance Reporting Initiatives SIM to Support Short and

409 views • 8 slides
NAHDO Data Quality Forum Cross State Metrics to Assess Data Quality September 2019 Identify

NAHDO Data Quality Forum Cross State Metrics to Assess Data Quality September 2019 Identify

NAHDO Data Quality Forum Cross State Metrics to Assess Data Quality September 2019 Identify cross-state metrics to compare data quality benchmarks Use NAHDOs Data Quality Forum to gather input and consensus on an agreed upon set of

275 views • 6 slides
1 QUALITY ASSURANCE - PROCESS QUALITY METRICS PREREQUISITES Top management commitment

1 QUALITY ASSURANCE - PROCESS QUALITY METRICS PREREQUISITES Top management commitment

SOFTWARE ENGINEERING SOFTWARE QUALITY - SOFTWARE QUALITY QUALITY COMPONENTS Today we talk about software process quality and Objective quality component: properties that can certification be measured or approximated objectively

237 views • 5 slides
Impact of particles on photolysis rates and Results Impact on photolysis air quality rates

Impact of particles on photolysis rates and Results Impact on photolysis air quality rates

Introduction Model description Impact of particles on photolysis rates and Results Impact on photolysis air quality rates Impact on concentrations Impact on air quality monitoring Elsa Real a and Karine Sartelet a Conclusions 3 juin

522 views • 20 slides
Metrics & Scoring Committee: Recommendations to Health Plan Quality Metrics Committee, 2019

Metrics & Scoring Committee: Recommendations to Health Plan Quality Metrics Committee, 2019

Metrics & Scoring Committee: Recommendations to Health Plan Quality Metrics Committee, 2019 Measure Set Will Brake, Committee Chair, AllCare Health Anna Jimenez, MD, Committee Vice-Chair, Family Care CCO November 9, 2017 HEALTH POLICY

191 views • 15 slides
Poverty and distributional impact of Covid-19 Crisis in Indonesia Arief Anshory Yusuf SDGs

Poverty and distributional impact of Covid-19 Crisis in Indonesia Arief Anshory Yusuf SDGs

Poverty and distributional impact of Covid-19 Crisis in Indonesia Arief Anshory Yusuf SDGs Center, Universitas Padjadjaran http://sdgcenter.unpad.ac.id Presented at How is COVID-19 changing development? WIDER Webinar Series, UNU/WIDER, 12

622 views • 21 slides
Can We Predict Scientific Impact with Social Media? A Comparison with Traditional Metrics of

Can We Predict Scientific Impact with Social Media? A Comparison with Traditional Metrics of

Can We Predict Scientific Impact with Social Media? A Comparison with Traditional Metrics of Scientific Impact Denis Helic Knowledge Technologies Institute, Graz University of Technology December 04, 2014 Helic (KTI) Scientific Impact

799 views • 52 slides
Distributional Compositionality Intro to Distributional Semantics Raffaella Bernardi University

Distributional Compositionality Intro to Distributional Semantics Raffaella Bernardi University

Distributional Compositionality Intro to Distributional Semantics Raffaella Bernardi University of Trento February 14, 2012 Acknowledgments Credits: Some of the slides of today lecture are based on earlier DS courses taught by Marco Baroni,

538 views • 49 slides
What we learned from Community Metrics Agenda Why are metrics used? How metrics are used

What we learned from Community Metrics Agenda Why are metrics used? How metrics are used

What we learned from Community Metrics Agenda Why are metrics used? How metrics are used in two Open Source communities Common pitfalls and ways to avoid them Why use Metrics? Transparency Who is contributing to the

166 views • 14 slides
The Quality Metrics Project Nick Merriman Manchester Museum Measuring quality & reach in

The Quality Metrics Project Nick Merriman Manchester Museum Measuring quality & reach in

The Quality Metrics Project Nick Merriman Manchester Museum Measuring quality & reach in arts Western Australian example ACE invited consortium of Manchester organisations to work with Intelligence Agency to explore potential of an

363 views • 19 slides
Development Metrics You Should Use (but dont) Development Metrics You Should Use (but

Development Metrics You Should Use (but dont) Development Metrics You Should Use (but

Development Metrics You Should Use (but dont) Development Metrics You Should Use (but Dont) @catswetel at #gotochgo @catswetel at #gotochgo QUALITY RESPONSIVENESS PRODUCTIVITY PREDICTABILITY QUALITY RESPONSIVENESS Troy Magennis

889 views • 67 slides
Software Metrics Denition The measurement of the software development process and its

Software Metrics Denition The measurement of the software development process and its

Object-Oriented Software Engineering Extra Chapter: Software Metrics Lecture 14 Why Measure Software? Projects often Go over budget Miss deadlines Exhibit poor quality Measurements can be used to improve the process and quality

526 views • 14 slides
Welcome! Measuring Impact: Lessons Learned Creating Alumni Engagement Metrics Only one rule!

Welcome! Measuring Impact: Lessons Learned Creating Alumni Engagement Metrics Only one rule!

Welcome! Measuring Impact: Lessons Learned Creating Alumni Engagement Metrics Only one rule! #1. Eliminate distractions, make data the hero! Creating an Alumni Engagement Metrics System 1. Advancement at the U 2. Project Purpose and Team 3.

571 views • 33 slides
Software Metrics Overview SE 350 Software Process & Product Quality Lecture Objectives

Software Metrics Overview SE 350 Software Process & Product Quality Lecture Objectives

Software Metrics Overview SE 350 Software Process & Product Quality Lecture Objectives Provide a survey of common software metrics Product metrics Project (process) metrics Use of product and project metrics in- process and

303 views • 17 slides
Defect Removal Metrics SE 350 Software Process & Product Quality 1 Objectives Understand

Defect Removal Metrics SE 350 Software Process & Product Quality 1 Objectives Understand

Defect Removal Metrics SE 350 Software Process & Product Quality 1 Objectives Understand some basic defect metrics and the concepts behind them Defect density metrics Defect detection and removal effectiveness etc. Look at

470 views • 28 slides
Assessment for Quality Education: Southeast Asia Primary Learning Metrics (SEA-PLM) What is

Assessment for Quality Education: Southeast Asia Primary Learning Metrics (SEA-PLM) What is

Assessment for Quality Education: Southeast Asia Primary Learning Metrics (SEA-PLM) What is SEA-PLM? The SEA-PLM serves the goal of improving quality of education through system level monitoring of learner achievements The initiative

497 views • 26 slides

More recommend