a pipeline for scalable text reuse analysis
play

A Pipeline for Scalable Text Reuse Analysis Milad Alshomary - PowerPoint PPT Presentation

Jun 11, 2023 •123 likes •1.01k views

A Pipeline for Scalable Text Reuse Analysis Milad Alshomary Bauhaus Universitt 05.07.2018 Milad Alshomary Pipeline for TR extraction 05.07.2018 1 Overview Motivation A Pipeline for Scalable Text Reuse Extraction Application on

  1. A Pipeline for Scalable Text Reuse Analysis Milad Alshomary Bauhaus Universität 05.07.2018 Milad Alshomary Pipeline for TR extraction 05.07.2018 1

  2. Overview ● Motivation ● A Pipeline for Scalable Text Reuse Extraction ● Application on Wikipedia ● Application on Wikipedia and Common Crawl ● Conclusion Milad Alshomary Pipeline for TR extraction 05.07.2018 2

  3. Motivation Text Reuse (TR) ● Quoting ● Verbatim ● Paraphrasing ● Translation ● Summarization Milad Alshomary Pipeline for TR extraction 05.07.2018 3

  4. Motivation TR Detection Applications Plagiarism detection METER project (Measuring Text Reuse) Milad Alshomary Pipeline for TR extraction 05.07.2018 4

  5. Motivation TR Detection Applications Plagiarism detection METER projet (Measuring Text Reuse) Milad Alshomary Pipeline for TR extraction 05.07.2018 5

  6. Motivation TR Detection Applications Plagiarism detection METER projet (Measuring Text Reuse) Milad Alshomary Pipeline for TR extraction 05.07.2018 6

  7. Motivation Wikipedia vs The World ● Digital Encyclopedia ● Collaborative environment ● Giant public source of information ● Free to use Milad Alshomary Pipeline for TR extraction 05.07.2018 7

  8. Motivation Wikipedia vs The World ● Digital Encyclopedia ● Collaborative environment ● Giant public source of information ● Free to use Milad Alshomary Pipeline for TR extraction 05.07.2018 8

  9. Motivation Wikipedia vs The World ● Digital Encyclopedia ● Collaborative environment ● Giant public source of information ● Free to use Milad Alshomary Pipeline for TR extraction 05.07.2018 9

  10. Motivation Wikipedia vs The World ● Digital Encyclopedia ● Collaborative environment ● Giant public source of information ● Free to use Milad Alshomary Pipeline for TR extraction 05.07.2018 10

  11. Motivation Wikipedia vs The World Quality Flaws Scientific community Milad Alshomary Pipeline for TR extraction 05.07.2018 11

  12. Motivation Wikipedia vs The World - Web pages = Wikipedia text + advertisements Milad Alshomary Pipeline for TR extraction 05.07.2018 12

  13. Motivation Research Questions ➔ What kinds of text reuse occur within Wikipedia? ➔ How much of the web is a copy of Wikipedia content? ➔ How much revenue does this content generate? Milad Alshomary Pipeline for TR extraction 05.07.2018 13

  14. Motivation Research Questions ➔ What kinds of text reuse occur within Wikipedia? ➔ How much of the web is a copy of Wikipedia content? ➔ How much revenue does this content generate? Milad Alshomary Pipeline for TR extraction 05.07.2018 14

  15. Motivation Research Questions ➔ What kinds of text reuse occur within Wikipedia? ➔ How much of the web is a copy of Wikipedia content? ➔ How much revenue does this content generate? Milad Alshomary Pipeline for TR extraction 05.07.2018 15

  16. A Pipeline for Scalable Text Reuse Extraction Milad Alshomary Pipeline for TR extraction 05.07.2018 16

  17. A Pipeline for Scalable Text Reuse Extraction Text Reuse Pipeline D1 TR Pipeline Input: Two datasets ➔ Output: Text reuse ➔ D2 cases Milad Alshomary Pipeline for TR extraction 05.07.2018 17

  18. A Pipeline for Scalable Text Reuse Extraction Text Reuse Pipeline D1 TR Pipeline Input: Two datasets ➔ Output: Text reuse ➔ D2 cases Milad Alshomary Pipeline for TR extraction 05.07.2018 18

  19. A Pipeline for Scalable Text Reuse Extraction Text Reuse Pipeline Text Candidate Text Preprocessing Elimination Alignment Content extraction ➔ Chunking ➔ Feature extraction ➔ Milad Alshomary Pipeline for TR extraction 05.07.2018 19

  20. A Pipeline for Scalable Text Reuse Extraction Text Reuse Pipeline Text Candidate Text Preprocessing Elimination Alignment Content extraction Pairwise scan ➔ ➔ Chunking Text Reuse heuristics ➔ ➔ Feature extraction ➔ Milad Alshomary Pipeline for TR extraction 05.07.2018 20

  21. A Pipeline for Scalable Text Reuse Extraction Text Reuse Pipeline Text Candidate Text Preprocessing Elimination Alignment Content extraction Pairwise scan Detailed scan of text ➔ ➔ ➔ Chunking Text Reuse heuristics reuse ➔ ➔ Feature extraction Picapica framework ➔ ➔ Milad Alshomary Pipeline for TR extraction 05.07.2018 21

  22. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination Text Candidate Text Preprocessing Elimination Alignment Keys for scaling-up: Cluster computing ➔ Heuristics based candidate elimination ➔ algorithms Milad Alshomary Pipeline for TR extraction 05.07.2018 22

  23. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination Text Candidate Text Preprocessing Elimination Alignment Keys for scaling-up: Cluster computing ➔ Heuristics based candidate elimination ➔ algorithms Milad Alshomary Pipeline for TR extraction 05.07.2018 23

  24. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination D 1 D 2 For a candidacy function we proposed the following methods: - Cosine similarity of TF-IDF (semantic) - Paragraph embedding (semantic) candidacy(d 11 , d 21 ) → [0, 1] - Stopwords N-grams (structure) - Weighted average of Stopwords Ngrams and d 11 d 21 Paragraph embedding (semantic + structure) d 22 d 12 d 2n d 1n Milad Alshomary Pipeline for TR extraction 05.07.2018 24

  25. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination Sample 1k documents Document Generate TR Sample from Wikipedia: Wikipedia Sample - Sample 1k documents from Wikipedia Text alignment using - Using Picapica framework to find picapica framework TR cases TR sample Milad Alshomary Pipeline for TR extraction 05.07.2018 25

  26. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination Sample 1k documents Document Generate TR Sample from Wikipedia: Wikipedia Sample - Sample 1k documents from Wikipedia Text alignment using - Using Picapica framework to find picapica framework TR cases TR sample Milad Alshomary Pipeline for TR extraction 05.07.2018 26

  27. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination Sample 1k documents Document Generate TR Sample from Wikipedia: Wikipedia Sample - Sample 1k documents from Wikipedia Text alignment using - Using Picapica framework to find picapica framework TR cases TR sample - 232 documents - ~ 90% have < 10 alignements (TR case) Milad Alshomary Pipeline for TR extraction 05.07.2018 27

  28. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination Evaluation of “ candidacy” function: TR sample - For each document in TR sample: - Sort all Wikipedia articles according to the proposed “ candidacy” . - Precision/Recall on Thresholds of [1, 101,..,100k] - A True Positive (TP) is a pair of documents that have TR. T1 T2 T3 Milad Alshomary Pipeline for TR extraction 05.07.2018 28

  29. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination Evaluation of “ candidacy” function: TR sample p1 p2 r2 r1 T1 T2 T3 Milad Alshomary Pipeline for TR extraction 05.07.2018 29

  30. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination Semantic hashing function: - Hashes documents into binary hashes. - Similar documents get similar or exact binary hash. 011001 011001 Milad Alshomary Pipeline for TR extraction 05.07.2018 30

  31. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination D 2 D 1 Semantic hashing function: - Hashing all documents. - Inverted index. - Hash document’s chunks. - Apply candidacy function only on Inverted index documents that intersect in one 001001 hash at least. 011001 011001 011001 001000 Milad Alshomary Pipeline for TR extraction 05.07.2018 31

  32. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination D 2 D 1 Semantic hashing function: - Hashing all documents. - Inverted index. - Hash document’s chunks. - Apply candidacy function only on Inverted index documents that intersect in one 001001 hash at least. 011001 011001 011001 001000 Milad Alshomary Pipeline for TR extraction 05.07.2018 32

  33. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination D 2 D 1 Semantic hashing function: - Hashing all documents. - Inverted index. - Hash document’s chunks. - Apply candidacy function only on Inverted index documents that intersect in one 001001 hash at least. 011001 011001 011001 001000 Milad Alshomary Pipeline for TR extraction 05.07.2018 33

  34. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination D 2 D 1 Semantic hashing function: - Hashing all documents. - Inverted index. - Hash document’s chunks. - Apply candidacy function only on Inverted index documents that intersect in one 001001 hash at least. 011001 011001 011001 001000 Milad Alshomary Pipeline for TR extraction 05.07.2018 34

  35. A Pipeline for Scalable Text Reuse Extraction Candidate Elimination d i Proposed semantic hashing methods: - Random Projection (data independent) - Variational Deep Semantic Hashing (data dependent) d j Milad Alshomary Pipeline for TR extraction 05.07.2018 35

Recommend

TRACER TUTORIAL: TEXT REUSE DETECTION INTRODUCTION TO HISTORICAL TEXT REUSE DETECTION M arco B

TRACER TUTORIAL: TEXT REUSE DETECTION INTRODUCTION TO HISTORICAL TEXT REUSE DETECTION M arco B

TRACER TUTORIAL: TEXT REUSE DETECTION INTRODUCTION TO HISTORICAL TEXT REUSE DETECTION M arco B uchler, Emily Franzini and Greta Franzini TABLE OF CONTENTS 1. Who am I? 2. What is text reuse? 3. Aspects of text reuse 4. ACID for the Digital

584 views • 34 slides
Orange County Water Reuse Peters Canyon Wash Channel Water Capture and Reuse Pipeline Project

Orange County Water Reuse Peters Canyon Wash Channel Water Capture and Reuse Pipeline Project

Orange County Water Reuse Peters Canyon Wash Channel Water Capture and Reuse Pipeline Project April 20, 2017 Agenda History and Approach Design and Construction Lessons and Challenges Questions 2 History and Approach Jian Peng, PhD

551 views • 26 slides
TRACER TUTORIAL: TEXT REUSE DETECTION RECENT WORK M arco B uchler, Emily Franzini and Greta

TRACER TUTORIAL: TEXT REUSE DETECTION RECENT WORK M arco B uchler, Emily Franzini and Greta

TRACER TUTORIAL: TEXT REUSE DETECTION RECENT WORK M arco B uchler, Emily Franzini and Greta Franzini METHODOLOGY Basic idea: Embed historical text reuse in Shannons N oisy Channel theorem. 2/26 MICROVIEW II Source: Stefan J anicke,

433 views • 26 slides
Text Reuse Detection Using a Composition of Text Similarity Measures Br, Zesch, Gurevych 2012

Text Reuse Detection Using a Composition of Text Similarity Measures Br, Zesch, Gurevych 2012

Text Reuse Detection Using a Composition of Text Similarity Measures Br, Zesch, Gurevych 2012 HS Computational study of Sabrina Galasso linguistic differences sabrina.galasso@student.uni-tuebingen.de December 16th, 2015 HS LingDiff 1 1.

1.46k views • 32 slides
Efficient Analysis of Pipeline Models for WCET Computation Stephan Wilhelm (sw@absint.com)

Efficient Analysis of Pipeline Models for WCET Computation Stephan Wilhelm (sw@absint.com)

A b s I n t Efficient Analysis of Pipeline Models for WCET Computation Stephan Wilhelm (sw@absint.com) AbsInt GmbH and Saarland University Outline 1. aiTs pipeline analysis 3. Why improve its efficiency? 5. BDD based pipeline analysis 7.

300 views • 10 slides
1 Infrastructure Requirements Limit Reuse Planned Indirect Potable Reuse (Purple pipe may be a

1 Infrastructure Requirements Limit Reuse Planned Indirect Potable Reuse (Purple pipe may be a

Overview of Presentation DIRECT POTABLE REUSE: A PATH FORWARD: Take Home Message Reuse Opportunities De Facto and Planned Indirect Potable Reuse 2012 WATER REUSE CONFERENCE Proposed Direct Potable Reuse Strategies Boise, ID

599 views • 8 slides
KnowledgeStore Scalable Framework for Interlinking Text and Knowledge Marco Rospocher

KnowledgeStore Scalable Framework for Interlinking Text and Knowledge Marco Rospocher

KnowledgeStore Scalable Framework for Interlinking Text and Knowledge Marco Rospocher Fondazione Bruno Kessler (FBK) rospocher@fbk.eu :: @marcorospocher KnowledgeStore A scalable, fault-tolerant, and Semantic Web grounded storage

695 views • 23 slides
Scalable Data Analytics Pipeline for Real-Time Attack Detection; Design, Validation, and

Scalable Data Analytics Pipeline for Real-Time Attack Detection; Design, Validation, and

Scalable Data Analytics Pipeline for Real-Time Attack Detection; Design, Validation, and Deployment in a Honeypot Environment Eric Badger Masters Student Computer Engineering 1 Overview Introduction/Motivation Challenges

508 views • 30 slides
ELECTRONIC TEXT REUSE ACQUISITION PROJECT INTRODUCTION &amp; MOTIVATION M arco Bchler TABLE OF

ELECTRONIC TEXT REUSE ACQUISITION PROJECT INTRODUCTION &amp; MOTIVATION M arco Bchler TABLE OF

ELECTRONIC TEXT REUSE ACQUISITION PROJECT INTRODUCTION &amp; MOTIVATION M arco Bchler TABLE OF CONTENTS 2/100 WHO AM I? WHO AM I? 2001-2002: Head of Quality Assurance department in a software company; 2006: Diploma in Computer

494 views • 23 slides
Social Media &amp; Text Analysis lecture 5 - POS/NE Tagging CSE 5539-0010 Ohio State University

Social Media &amp; Text Analysis lecture 5 - POS/NE Tagging CSE 5539-0010 Ohio State University

Social Media &amp; Text Analysis lecture 5 - POS/NE Tagging CSE 5539-0010 Ohio State University Instructor: Alan Ritter Website: socialmedia-class.org NLP Pipeline (summary so far) classification Regular (Nave Bayes) Expression Part-of-

1.17k views • 60 slides
Scalable Full-Text Search for Petascale File Systems Andrew W. Leung Ethan L. Miller

Scalable Full-Text Search for Petascale File Systems Andrew W. Leung Ethan L. Miller

Scalable Full-Text Search for Petascale File Systems Andrew W. Leung Ethan L. Miller University of California, Santa Cruz 3 rd Petascale Data Storage Workshop (PDSW 08) November 17th, 2008 Need scalable file management Todays

307 views • 12 slides
KnowledgeStore Scalable Framework for Interlinking Text and Knowledge Marco Rospocher,

KnowledgeStore Scalable Framework for Interlinking Text and Knowledge Marco Rospocher,

KnowledgeStore Scalable Framework for Interlinking Text and Knowledge Marco Rospocher, Bernardo Magnini, Luciano Serafini Fondazione Bruno Kessler (FBK) INTRODUCTION Information is typically available both in unstructured and

707 views • 43 slides
Scalable Data Analytics Pipeline for Validation of Real-Time Attack Detection Eric Badger , Phuong

Scalable Data Analytics Pipeline for Validation of Real-Time Attack Detection Eric Badger , Phuong

Scalable Data Analytics Pipeline for Validation of Real-Time Attack Detection Eric Badger , Phuong Cao, Alex Withers, Adam Slagell, Zbigniew Kalbarczyk, Ravi Iyer University of Illinois Urbana-Champaign 1 Overview Introduction/Motivation

283 views • 24 slides
Building Scalable and Extendable Data Pipeline for Call of Duty Games: Lessons Learned Yaroslav

Building Scalable and Extendable Data Pipeline for Call of Duty Games: Lessons Learned Yaroslav

Building Scalable and Extendable Data Pipeline for Call of Duty Games: Lessons Learned Yaroslav Tkachenko Senior Data Engineer at Activision 1+ Data lake size (AWS S3) PB Number of topics in the biggest cluster 500+ (Apache Kafka)

548 views • 44 slides
Text Mining in Hebrew Impact of Morphology Analysis on Topic Analysis and on Search Quality

Text Mining in Hebrew Impact of Morphology Analysis on Topic Analysis and on Search Quality

Text Mining in Hebrew Impact of Morphology Analysis on Topic Analysis and on Search Quality Michael Elhadad, Meni Adler, Yoav Goldberg, Dudi Gabay and Yael Netzer Text in Hebrew Extract information from text in Hebrew Major immediate

1k views • 88 slides
Introduction to Historical Text Reuse Detection Marco Bchler, Emily Franzini, Greta Franzini,

Introduction to Historical Text Reuse Detection Marco Bchler, Emily Franzini, Greta Franzini,

Introduction to Historical Text Reuse Detection Marco Bchler, Emily Franzini, Greta Franzini, Maria Moritz eTRAP Research Group Gttingen Centre for Digital Humanities Institute of Computer Science Georg August University Gttingen, Germany

714 views • 25 slides
Building Scalable Real-Time Data Pipeline Data Fridge Vicente Valls Rios Software

Building Scalable Real-Time Data Pipeline Data Fridge Vicente Valls Rios Software

Building Scalable Real-Time Data Pipeline Data Fridge Vicente Valls Rios Software Engineering Manager !!! 2 million orders delivered in one day (July 2019) www.deliveryhero.com Data Volume in Numbers + User clicks + Logistics data

283 views • 27 slides
ELECTRONIC TEXT REUSE ACQUISITION PROJECT INTRODUCTION &amp; MOTIVATION M arco Bchler, Greta

ELECTRONIC TEXT REUSE ACQUISITION PROJECT INTRODUCTION &amp; MOTIVATION M arco Bchler, Greta

ELECTRONIC TEXT REUSE ACQUISITION PROJECT INTRODUCTION &amp; MOTIVATION M arco Bchler, Greta Franzini, Emily Franzini TABLE OF CONTENTS 1. Who are we? 2. Motivation 2/16 WHO ARE WE? WHO AM I? 2001-2002: Head of Quality Assurance

293 views • 16 slides
TRACER TUTORIAL: TEXT REUSE DETECTION SELECTION Mar co B uchler, Emily Franzini and Greta

TRACER TUTORIAL: TEXT REUSE DETECTION SELECTION Mar co B uchler, Emily Franzini and Greta

TRACER TUTORIAL: TEXT REUSE DETECTION SELECTION Mar co B uchler, Emily Franzini and Greta Franzini TABLE OF CONTENTS 1. Wha t is Selection? 2. Selection techniques 3. Hacking 4. Conclusion and revision 2/29 REMINDER: CURRENT APPROACH 3/29

321 views • 29 slides
Software Reuse From informal reuse (scavenging) to systematic reuse Management and technical

Software Reuse From informal reuse (scavenging) to systematic reuse Management and technical

Software Reuse From informal reuse (scavenging) to systematic reuse Management and technical issues Michal Young, SERC 05/19/99 1 Merry-Go-Round of Sequential Development Conventional view of development: Requirements Common theme -

238 views • 13 slides
Big Data: Pipeline Demo Day Analysis of white matter shapes Nic Novak NSIDP 2 nd Year,

Big Data: Pipeline Demo Day Analysis of white matter shapes Nic Novak NSIDP 2 nd Year,

Big Data: Pipeline Demo Day Analysis of white matter shapes Nic Novak NSIDP 2 nd Year, Laboratory of Neuroimaging Summary White matter morphology and Alzheimers LONI Pipeline / methodology Results A problem for Pipeline

163 views • 14 slides
TRACER TUTORIAL: TEXT REUSE DETECTION FEATURING M arco B uchler, Emily Franzini and Greta

TRACER TUTORIAL: TEXT REUSE DETECTION FEATURING M arco B uchler, Emily Franzini and Greta

TRACER TUTORIAL: TEXT REUSE DETECTION FEATURING M arco B uchler, Emily Franzini and Greta Franzini TABLE OF CONTENTS 1. What is featuring? 2. Featuring techniques 3. Hacking 4. Conclusion and revision 2/27 REMINDER: CURRENT APPROACH 3/27

461 views • 27 slides
TRACER TUTORIAL: TEXT REUSE DETECTION INTRODUCTION TO THE COMMAND LINE AND ACCESSING SERVERS

TRACER TUTORIAL: TEXT REUSE DETECTION INTRODUCTION TO THE COMMAND LINE AND ACCESSING SERVERS

TRACER TUTORIAL: TEXT REUSE DETECTION INTRODUCTION TO THE COMMAND LINE AND ACCESSING SERVERS REMOTELY M arco B uchler, Emily Franzini and Greta Franzini TABLE OF CONTENTS 1. C onnecting to the server 2. Command line introduction 2/100

679 views • 19 slides
Alias Analysis Last time Reuse optimization Today Alias analysis (pointer analysis)

Alias Analysis Last time Reuse optimization Today Alias analysis (pointer analysis)

Alias Analysis Last time Reuse optimization Today Alias analysis (pointer analysis) Next time More alias analysis (pointer analysis) CS553 Lecture Alias Analysis I 2 Aliasing What is aliasing? When two expressions denote

265 views • 8 slides

More recommend