understanding task design trade offs in crowdsourced
play

Understanding Task Design Trade-offs in Crowdsourced Paraphrase - PowerPoint PPT Presentation

Feb 10, 2023 •149 likes •624 views

Understanding Task Design Trade-offs in Crowdsourced Paraphrase Collection Youxuan Lucy Jiang, Jonathan Kummerfeld and Walter Lasecki Paraphrases are useful across Motivation Experiment a wide range of NLP tasks Results

  1. Understanding Task Design Trade-offs in Crowdsourced Paraphrase Collection Youxuan Lucy Jiang, Jonathan Kummerfeld and 
 Walter Lasecki

  2. Paraphrases are useful across 
 Motivation 
 Experiment 
 a wide range of NLP tasks Results

  3. 
 
 
 Paraphrases are useful across 
 Motivation 
 Experiment 
 a wide range of NLP tasks Results Sentence 
 Paraphrase 1 
 Paraphrase 2 
 …

  4. 
 
 
 Paraphrases are useful across 
 Motivation 
 Experiment 
 a wide range of NLP tasks Results Sentence 
 Paraphrase 1 
 Paraphrase 2 
 …

  5. 
 
 
 Paraphrases are useful across 
 Motivation 
 Experiment 
 a wide range of NLP tasks Results Sentence 
 Paraphrase 1 
 Structured Output Paraphrase 2 
 …

  6. Paraphrases are useful across 
 Motivation 
 Experiment 
 a wide range of NLP tasks Results The boy desires the girl 
 to believe him. The boy desires to be 
 believed by the girl. The boy has a desire to 
 be believed by the girl. The boy’s desire is for 
 the girl to believe him. …

  7. 
 
 
 Paraphrases are useful across 
 Motivation 
 Experiment 
 a wide range of NLP tasks Results What is the smallest 
 state by area? 
 λ x1.smallest(x2, 
 Which state has 
 λ x1.state(x1), 
 the least area? 
 λ x1. λ x2.area(x1,x2)) Name the state with 
 the smallest area 
 …

  8. 
 
 
 Paraphrases are useful across 
 Motivation 
 Experiment 
 a wide range of NLP tasks Results What is the smallest 
 state by area? 
 SELECT state.state_name 
 FROM state 
 Which state has 
 WHERE state.area = ( the least area? 
 SELECT MIN (state.area) FROM state Name the state with 
 ) ; the smallest area 
 …

  9. 
 
 
 Paraphrases are useful across 
 Motivation 
 Experiment 
 a wide range of NLP tasks Results What is the smallest 
 state by area? 
 SELECT state.state_name 
 FROM state 
 Which state has 
 WHERE state.area = ( the least area? 
 SELECT MIN (state.area) FROM state Name the state with 
 ) ; the smallest area 
 …

  10. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results

  11. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results

  12. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results Title

  13. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results Title Instructions …

  14. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results Title Instructions … Examples

  15. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results Title Instructions … Examples Prompt Sentence

  16. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results Title Instructions … Examples Prompt Sentence

  17. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results Title Instructions … Examples Prompt Sentence

  18. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results Title Instructions … Examples Prompt Sentence

  19. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results Title Instructions … Examples Prompt Sentence

  20. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results Title Instructions … Examples Prompt Sentence

  21. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results Title Instructions … $0.05 - participation Examples Prompt Sentence

  22. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results Title Instructions … $0.05 - participation Examples $0.05 - bonus Prompt Sentence

  23. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results Title Instructions … Examples Prompt Sentence

  24. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results

  25. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results

  26. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results

  27. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results

  28. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results

  29. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results

  30. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results

  31. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results

  32. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results

  33. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results

  34. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results

  35. Motivation 
 Experiment 
 Results

  36. Motivation 
 Experiment 
 Results 100 90 80 Correctness 70 Diversity 60 50 Syntactic Lexical 
 Lexical & 
 No Examples Variation Syntactic

  37. Motivation 
 Experiment 
 Results 100 90 80 Correctness 70 Diversity 60 50 Syntactic Lexical 
 Lexical & 
 No Examples Variation Syntactic

  38. Motivation 
 Experiment 
 Results 100 90 80 Correctness 70 Diversity 60 50 Syntactic Lexical 
 Lexical & 
 No Examples Variation Syntactic

  39. Motivation 
 Priming has a significant impact Experiment 
 Results 100 90 80 Correctness 70 Diversity 60 50 Syntactic Lexical 
 Lexical & 
 No Examples Variation Syntactic

  40. Motivation 
 Priming has a significant impact Experiment 
 Results 100 90 80 Correctness 70 Diversity 60 50 Syntactic Lexical 
 Lexical & 
 No Examples Variation Syntactic

  41. Motivation 
 Priming has a significant impact Experiment 
 Results 100 90 80 Correctness 70 Diversity 60 50 Single 
 Chain Answers Source

  42. Motivation 
 Priming has a significant impact Experiment 
 Results 100 90 80 Correctness 70 Diversity 60 50 Single 
 Chain Answers Source

  43. Motivation 
 Priming has a significant impact Experiment 
 Results 100 90 80 Correctness 70 Diversity 60 50 Single 
 Chain Answers Source

  44. Motivation 
 Jargon is a challenge Experiment 
 Results 100 100 90 80 70 Correctness 60 Diversity 50 40 30 30 Course 
 Advising 
 GeoQuery WSJ Ubuntu 
 Questions Dialogue IRC

  45. Motivation 
 Jargon is a challenge Experiment 
 Results 100 100 90 80 70 Correctness 60 Diversity 50 40 30 30 Course 
 Advising 
 GeoQuery WSJ Ubuntu 
 Questions Dialogue IRC

  46. Motivation 
 Jargon is a challenge Experiment 
 Results 100 100 90 80 70 Correctness 60 Diversity 50 40 30 30 Course 
 Advising 
 GeoQuery WSJ Ubuntu 
 Questions Dialogue IRC

  47. The first systematic study of trade-offs Motivation 
 Experiment 
 when crowdsourcing paraphrases Results Thank you! Youxuan Lucy Jiang, Jonathan Kummerfeld and 
 Walter Lasecki 
 Supported by IBM

Recommend

TRADE-OFFS AMONG AI TRADE-OFFS AMONG AI TECHNIQUES TECHNIQUES Christian Kaestner With slides

TRADE-OFFS AMONG AI TRADE-OFFS AMONG AI TECHNIQUES TECHNIQUES Christian Kaestner With slides

TRADE-OFFS AMONG AI TRADE-OFFS AMONG AI TECHNIQUES TECHNIQUES Christian Kaestner With slides adopted from Eunsuk Kang Required reading: Vogelsang, Andreas, and Markus Borg. " Requirements Engineering for Machine Learning:

760 views • 54 slides
Chapter 2 Trade-offs, Comparative Advantage, and the Market System Modeling Trade-offs:

Chapter 2 Trade-offs, Comparative Advantage, and the Market System Modeling Trade-offs:

Chapter 2 Trade-offs, Comparative Advantage, and the Market System Modeling Trade-offs: Production Possibilities Frontier (PPF) The PPF: Demonstrates the maximum attainable combination of two products with a given technology and a given

592 views • 13 slides
Markov chaining Checkout DataStructures project from SVN Understanding the engineering trade-offs

Markov chaining Checkout DataStructures project from SVN Understanding the engineering trade-offs

Data-structure-palooza Fixed-length queues Markov chaining Checkout DataStructures project from SVN Understanding the engineering trade-offs when storing data Boil down data types (e.g., lists) to their essential operations Choosing a

602 views • 20 slides
Checkout DataStructures from SVN Understanding the engineering trade-offs when storing data

Checkout DataStructures from SVN Understanding the engineering trade-offs when storing data

Data-structure-palooza Checkout DataStructures from SVN Understanding the engineering trade-offs when storing data Boil down data types (e.g., lists) to their essential operations Choosing a data structure for a project then becomes:

664 views • 14 slides
Data Structures + BIG-O Notation Understanding the engineering trade-offs when storing data

Data Structures + BIG-O Notation Understanding the engineering trade-offs when storing data

CSSE 220 Data Structures + BIG-O Notation Understanding the engineering trade-offs when storing data Checkout LinkedListSimple project from public git Checkout SinglyLinkedList homework from git Data Structures Efficient ways to store data

239 views • 9 slides
Energy-efficient Computing for Wildlife Tracking Design Trade-offs and Early Experiences with

Energy-efficient Computing for Wildlife Tracking Design Trade-offs and Early Experiences with

Energy-efficient Computing for Wildlife Tracking Design Trade-offs and Early Experiences with ZebraNet Philo J uang, Hidekazu Oki Yong Wang, Margaret Martonosi Li-Shiuan Peh, Dan Rubenstein Princeton University CS Z25/4C38: Mobile and

503 views • 37 slides
Trade Offs in the Design of a Router with Both Guaranteed and Best-Effort Services for Networks

Trade Offs in the Design of a Router with Both Guaranteed and Best-Effort Services for Networks

Trade Offs in the Design of a Router with Both Guaranteed and Best-Effort Services for Networks on Chip E. Rijpkema, K. Goossens, A. R dulescu, J. Dielissen, J. van Meerbergen, P. Wielage, and E. Waterlander why Networks-on-Chip problems

523 views • 23 slides
Design and Performance Trade-offs in Parallelized RF SDR Architecture Ville Saari 1 arssinen 2

Design and Performance Trade-offs in Parallelized RF SDR Architecture Ville Saari 1 arssinen 2

Design and Performance Trade-offs in Parallelized RF SDR Architecture Ville Saari 1 arssinen 2 Sami Kiminki Aarno P Antti Immonen 2 Vesa Hirvisalo Jussi Ryyn anen Tommi Zetterman Aalto University Aalto University Nokia Research Center

600 views • 17 slides
History of Operating Systems What drives these trade-offs? Hardware User Applications

History of Operating Systems What drives these trade-offs? Hardware User Applications

10/1/16 COMP 530: Operating Systems COMP 530: Operating Systems Natural Selection Almost all OS design is about trade-offs History of Operating Systems What drives these trade-offs? Hardware User Applications Observation:

381 views • 6 slides
Time-memory Trade-offs for Near-collisions Conclusion Combining trunc & codes Time-memory

Time-memory Trade-offs for Near-collisions Conclusion Combining trunc & codes Time-memory

Introduction 1/24 Gatan Leurent Time-memory Trade-offs for Near-collisions Conclusion Combining trunc & codes Time-memory trade-offs Memoryless FSE 2013 UCL Crypto Group FSE 2013 Time-memory Trade-offs for Near-collisions G.

430 views • 31 slides
CrowdQ: Crowdsourced Query Understanding Gianluca Demar8ni, Beth

CrowdQ: Crowdsourced Query Understanding Gianluca Demar8ni, Beth

CrowdQ: Crowdsourced Query Understanding Gianluca Demar8ni, Beth Trushkowsky, Tim Kraska, Michael J. Franklin Scenario Find the birthdate of the mayor of

316 views • 15 slides
Extreme-scale Data Resilience Trade-offs at Experimental Facilities Sadaf Alam Chief

Extreme-scale Data Resilience Trade-offs at Experimental Facilities Sadaf Alam Chief

Extreme-scale Data Resilience Trade-offs at Experimental Facilities Sadaf Alam Chief Technology O ffi cer Swiss National Supercomputing Centre MSST (May 22, 2019) Outline Background Users, customers and services Co-design,

441 views • 25 slides
Enemy targeting, trade - offs, and the evolutionary Assembly of

Enemy targeting, trade - offs, and the evolutionary Assembly of

Enemy targeting, trade - offs, and the evolutionary Assembly of a tortoise beetle defense arsenal Fredric V. Vencl Ecology and Evolution, Stony Brook

262 views • 9 slides
Filling in the Blanks in Understanding Discourse Adverbials: Consistency, Conflict,

Filling in the Blanks in Understanding Discourse Adverbials: Consistency, Conflict,

Filling in the Blanks in Understanding Discourse Adverbials: Consistency, Conflict, and Context-Dependence in a Crowdsourced Elicitation Task Hannah Rohde, Anna Dickinson, Nathan Schneider, Christopher N. L. Clark, Annie Louis, &

818 views • 23 slides
Understand the trade-offs using compilers for Java applications (From AOT to JIT and Beyond!)

Understand the trade-offs using compilers for Java applications (From AOT to JIT and Beyond!)

Understand the trade-offs using compilers for Java applications (From AOT to JIT and Beyond!) Mark Stoodley Eclipse OpenJ9 & OMR project co-lead Senior Software Developer @ IBM Canada mstoodle@ca.ibm.com @mstoodle Java ecosystem has a

827 views • 56 slides
Object/Class Design Slides adapted from Craig Zilles 1 Design is hard Design is an art, not a

Object/Class Design Slides adapted from Craig Zilles 1 Design is hard Design is an art, not a

Object/Class Design Slides adapted from Craig Zilles 1 Design is hard Design is an art, not a science Large/infinite design space, not enumerable Requirements, constraints, trade-offs, and priorities You get better with practice /

417 views • 15 slides
I/O Virtualization with Hardware Support Ubaid H. and Vasia P. Trade-offs and Motivation - In

I/O Virtualization with Hardware Support Ubaid H. and Vasia P. Trade-offs and Motivation - In

I/O Virtualization with Hardware Support Ubaid H. and Vasia P. Trade-offs and Motivation - In a fully virtual I/O, we have interposition - Full encapsulation (store, pause, resume), portability, flexibility, live migration - Slow

423 views • 24 slides
SociableSense: Exploring the Trade- offs of Adaptive Sampling and Computation Offloading for

SociableSense: Exploring the Trade- offs of Adaptive Sampling and Computation Offloading for

SociableSense: Exploring the Trade- offs of Adaptive Sampling and Computation Offloading for Social Sensing Kiran K. Rachuri, Cecilia Mascolo, Mirco Musolesi, Peter J. Rentfrow Mateusz Grabowski The images come from an original presentation:

373 views • 33 slides
Privacy Trade Offs of Geo Location General Population Concerns and an Application to

Privacy Trade Offs of Geo Location General Population Concerns and an Application to

Privacy Trade Offs of Geo Location General Population Concerns and an Application to the 2020 US Census Laura Brandimarte Alessandro Acquisti 1 2 How does this affect willingness to disclose personal information? 3 What is the

682 views • 30 slides
Efficiency and Distributional Trade-Offs in Recycling Carbon Cap-and-Trade Revenues Ian Parry

Efficiency and Distributional Trade-Offs in Recycling Carbon Cap-and-Trade Revenues Ian Parry

Efficiency and Distributional Trade-Offs in Recycling Carbon Cap-and-Trade Revenues Ian Parry Resources For the Future Roberton C. Williams III University of Maryland, University of Texas, Resources For the Future, and NBER Distributional

693 views • 15 slides
The Theory of Web Transparency: Algorithms and Trade-offs Augustin Chaintreau 1 , Guillaume

The Theory of Web Transparency: Algorithms and Trade-offs Augustin Chaintreau 1 , Guillaume

The Theory of Bringing Privacy into Practice, 2015 1/8 The Theory of Web Transparency: Algorithms and Trade-offs Augustin Chaintreau 1 , Guillaume Ducoffe 2 , 3 , Roxana Geambasu 1 , ecuyer 1 Mathias L 1Columbia University 2Univ. Nice Sophia

339 views • 14 slides
Distributed frequency allocation algorithms for cellular networks: Trade-offs and tuning

Distributed frequency allocation algorithms for cellular networks: Trade-offs and tuning

Distributed frequency allocation algorithms for cellular networks: Trade-offs and tuning strategies Marina Papatriantafilou, David Rutter and Philippas Tsigas Chalmers University of Technology Gothenburg, Sweden. What did we do ?

376 views • 18 slides
1 Typical design trade-offs System design Functionality Usability 1 Overview Cost

1 Typical design trade-offs System design Functionality Usability 1 Overview Cost

System design Chair of Softw are Engineering Software Engineering 1 Overview Prof. Dr. Bertrand Meyer 2 Subsystem decomposition March 2007 June 2007 3 Assessing O-O architectures 4 Architectural styles Lecture 10: 5

692 views • 24 slides
Trade-offs between regionally equitable and cost-efficient allocations of renewable electricity

Trade-offs between regionally equitable and cost-efficient allocations of renewable electricity

Trade-offs between regionally equitable and cost-efficient allocations of renewable electricity generation Jan-Philipp Sasse, Evelina Trutnevyte Renewable Energy Systems Group, University of Geneva 16 th IAEE European Conference 2019 | Ljubljana,

769 views • 28 slides

More recommend