a machine learning perspective on managing noisy data
play

A Machine Learning Perspective on Managing Noisy Data Theodoros - PowerPoint PPT Presentation

Dec 03, 2022 •338 likes •1.03k views

A Machine Learning Perspective on Managing Noisy Data Theodoros Rekatsinas | UW-Madison @thodrek Data-hungry applications are taking over Data errors are everywhere Noisy measurements Sensor failures Data errors are everywhere

  1. A Machine Learning Perspective on Managing Noisy Data Theodoros Rekatsinas | UW-Madison @thodrek

  2. Data-hungry applications are taking over

  3. Data errors are everywhere • Noisy measurements • Sensor failures

  4. Data errors are everywhere • Uncertain extractions • Semantic ambiguity

  5. Data errors are everywhere • Adversarial examples

  6. Data errors are everywhere • Human errors • Machine failures • Code bugs

  7. The Achilles’ Heel of Modern Analytics is low quality, erroneous data

  8. The Achilles’ Heel of Modern Analytics is low quality, erroneous data Cleaning and organizing the data comprises 60% of the time spent on an analytics or AI project.

  9. The Achilles’ Heel of Modern Analytics is low quality, erroneous data Many modern data management systems are being developed to address aspects of this issue: Stanford’s Snorkel: A System for Fast Training Data Creation Google’s TFX: TensorFlow Data Validation Amazon’s SageMaker Amazon’s Deequ: Data Quality Validation for ML Pipelines HoloClean: Weakly-supervised data cleaning

  10. Question: What is an appropriate (formal) framework for managing noisy data? Things to consider: Simplicity and generality

  11. Talk outline • Managing Noisy Data (Background) • The Probabilistic Unclean Databases (PUDs) Framework • From Theory to Systems

  12. Managing Noisy Data

  13. A simple example of noisy data DBAName AKAName Address City State Zip 3465 S Chicago 60608 t1 John Veliotis Sr. Johnnyo’s IL Morgan ST Conflicts 3465 S 60609 t2 John Veliotis Sr. Johnnyo’s Chicago IL Morgan ST 3465 S 60609 t3 John Veliotis Sr. Johnnyo’s Chicago IL c1: DBAName → Zip Morgan ST c2: Zip → City, State 3465 S Johnnyo’s Cicago t4 Johnnyo’s IL 60608 Morgan ST c3: City, State, Address → Zip Does not obey Conflict data distribution

  14. A simple example of noisy data DBAName AKAName Address City State Zip 3465 S Chicago 60608 t1 John Veliotis Sr. Johnnyo’s IL Morgan ST Conflicts 3465 S 60609 t2 John Veliotis Sr. Johnnyo’s Chicago IL Morgan ST 3465 S 60609 t3 John Veliotis Sr. Johnnyo’s Chicago IL c1: DBAName → Zip Morgan ST c2: Zip → City, State 3465 S Johnnyo’s Cicago t4 Johnnyo’s IL 60608 Morgan ST c3: City, State, Address → Zip Does not obey Conflict data distribution Computational problems: Detect errors, repair errors, compute “ consistent” query answers .

  15. The case for inconsistent data DBAName AKAName Address City State Zip 3465 S Chicago 60608 t1 John Veliotis Sr. Johnnyo’s IL Morgan ST c1: DBAName → Zip 3465 S 60609 t2 John Veliotis Sr. Johnnyo’s Chicago IL c2: Zip → City, State Morgan ST c3: City, State, Address → Zip 3465 S 60609 t3 John Veliotis Sr. Johnnyo’s Chicago IL Morgan ST 3465 S Johnnyo’s Cicago t4 Johnnyo’s IL 60608 Morgan ST An example unclean database J • Errors correspond to tuples/cells that introduce inconsistencies (violations of integrity constraints). • Inconsistencies are typical in data integration, extract-load-transform workloads, etc. • Data repairs: A theoretical framework for coping with inconsistent databases [Arenas et al. 1999]

  16. Minimal data repairs Slide by Phokion Kolaitis [SAT 2016]

  17. Minimal data repairs Plethora of fundamental results on tractability of repair-checking and consistent query answering. Slide by Phokion Kolaitis [SAT 2016]

  18. Minimal data repairs Plethora of fundamental results on tractability of repair-checking and consistent query answering. Limited adoption in practice. Slide by Phokion Kolaitis [SAT 2016]

  19. Minimal data repairs c1: DBAName → Zip DBAName AKAName Address City State Zip c2: Zip → City, State 3465 S Chicago 60608 t1 John Veliotis Sr. Johnnyo’s IL c3: City, State, Address → Zip Morgan ST 3465 S 60609 t2 John Veliotis Sr. Johnnyo’s Chicago IL Morgan ST 3465 S 60609 t3 John Veliotis Sr. Johnnyo’s Chicago IL Morgan ST 3465 S Johnnyo’s Cicago t4 Johnnyo’s IL 60608 Morgan ST

  20. Minimal data repairs c1: DBAName → Zip DBAName AKAName Address City State Zip c2: Zip → City, State 3465 S Chicago 60608 t1 John Veliotis Sr. Johnnyo’s IL c3: City, State, Address → Zip Morgan ST 3465 S 60609 t2 John Veliotis Sr. Johnnyo’s Chicago IL Morgan ST Minimal subset repair: 3465 S 60609 t3 John Veliotis Sr. Johnnyo’s Chicago IL We remove t1 Morgan ST 3465 S Johnnyo’s Cicago t4 Johnnyo’s IL 60608 Morgan ST An example repaired database I

  21. Minimal data repairs c1: DBAName → Zip DBAName AKAName Address City State Zip c2: Zip → City, State 3465 S Chicago 60608 t1 John Veliotis Sr. Johnnyo’s IL c3: City, State, Address → Zip Morgan ST 3465 S 60609 t2 John Veliotis Sr. Johnnyo’s Chicago IL Morgan ST Minimal subset repair: 3465 S 60609 t3 John Veliotis Sr. Johnnyo’s Chicago IL We remove t1 Morgan ST 3465 S Johnnyo’s Cicago t4 Johnnyo’s IL 60608 Morgan ST Errors remain: (1) Cicago should clearly be Chicago (2) Non-obvious errors: 60609 is the wrong Zip

  22. Minimal data repairs c1: DBAName → Zip DBAName AKAName Address City State Zip c2: Zip → City, State 3465 S Chicago 60608 t1 John Veliotis Sr. Johnnyo’s IL c3: City, State, Address → Zip Morgan ST 3465 S 60609 t2 John Veliotis Sr. Johnnyo’s Chicago IL Morgan ST Minimal subset repair: 3465 S 60609 t3 John Veliotis Sr. Johnnyo’s Chicago IL We remove t1 Morgan ST 3465 S Johnnyo’s Cicago t4 Johnnyo’s IL 60608 Several variations of Morgan ST minimal repairs. E.g., Errors remain: update the minimum (1) Cicago should clearly be Chicago number of cells. (2) Non-obvious errors: 60609 is the wrong Zip

  23. Minimal data repairs c1: DBAName → Zip DBAName AKAName Address City State Zip c2: Zip → City, State 3465 S Chicago 60608 t1 John Veliotis Sr. Johnnyo’s IL c3: City, State, Address → Zip Morgan ST 3465 S 60609 t2 John Veliotis Sr. Johnnyo’s Chicago IL Morgan ST Minimal subset repair: 3465 S 60609 t3 John Veliotis Sr. Johnnyo’s Chicago IL We remove t1 Morgan ST 3465 S Johnnyo’s Cicago t4 Johnnyo’s IL 60608 Several variations of Morgan ST minimal repairs. E.g., Errors remain: update the minimum (1) Cicago should clearly be Chicago number of cells. (2) Non-obvious errors: 60609 is the wrong Zip Minimality can be used as an operational principle to prioritize repairs but these repairs are not necessarily correct with respect to the ground truth.

  24. The case for most probable data [Gribkoff et al., 14] DBAName AKAName Address City State Zip p 3465 S Chicago 60608 0.9 t1 John Veliotis Sr. Johnnyo’s IL Morgan ST 3465 S 0.4 60609 t2 John Veliotis Sr. Johnnyo’s Chicago IL Morgan ST 3465 S 0.4 60609 t3 John Veliotis Sr. Johnnyo’s Chicago IL Morgan ST 3465 S 0.8 Johnnyo’s Cicago t4 Johnnyo’s IL 60608 Morgan ST Most probable world, c1: DBAName → Zip conditioned on integrity c2: Zip → City, State constraint satisfaction c3: City, State, Address → Zip

  25. The case for most probable data [Gribkoff et al., 14] DBAName AKAName Address City State Zip p Factor (f) 3465 S Chicago 60608 0.9 1 - 0.9 t1 John Veliotis Sr. Johnnyo’s IL Morgan ST 3465 S 0.4 0.4 60609 t2 John Veliotis Sr. Johnnyo’s Chicago IL Morgan ST 3465 S 0.4 0.4 60609 t3 John Veliotis Sr. Johnnyo’s Chicago IL Morgan ST 3465 S 0.8 0.8 Johnnyo’s Cicago t4 Johnnyo’s IL 60608 Morgan ST Optimization Objective c1: DBAName → Zip c2: Zip → City, State ∏ p ( t ) ∏ (1 − p ( t )) c3: City, State, Address → Zip max I t ∈ I t ∉ I

  26. The case for most probable data [Gribkoff et al., 14] DBAName AKAName Address City State Zip p Factor (f) 3465 S Chicago 60608 0.9 0.9 t1 John Veliotis Sr. Johnnyo’s IL Morgan ST 3465 S 0.4 1 - 0.4 60609 t2 John Veliotis Sr. Johnnyo’s Chicago IL Morgan ST 3465 S 0.4 1 - 0.4 60609 t3 John Veliotis Sr. Johnnyo’s Chicago IL Morgan ST 3465 S 0.8 1 - 0.8 Johnnyo’s Cicago t4 Johnnyo’s IL 60608 Morgan ST Optimization Objective c1: DBAName → Zip c2: Zip → City, State ∏ p ( t ) ∏ (1 − p ( t )) c3: City, State, Address → Zip max I t ∈ I t ∉ I

  27. Most probable repairs DBAName AKAName Address City State Zip p Factor (f) 3465 S Chicago 60608 0.9 1 - 0.9 t1 John Veliotis Sr. Johnnyo’s IL Morgan ST 3465 S 0.4 0.4 60609 t2 John Veliotis Sr. Johnnyo’s Chicago IL Morgan ST 3465 S 0.4 0.4 60609 t3 John Veliotis Sr. Johnnyo’s Chicago IL Morgan ST 3465 S 0.8 0.8 Johnnyo’s Cicago t4 Johnnyo’s IL 60608 Morgan ST ∏ p ( t ) ∏ Optimization Objective (1 − p ( t )) max I t ∈ I t ∉ I Probabilities o ff er clear semantics than minimality. Fundamental question: How do we know p?

  28. Probabilistic Unclean Databases Christopher De Sa, Ihab Ilyas, Benny Kimelfeld, Christopher Ré, Theodoros Rekatsinas, ICDT 2019

  29. ̂ The case of a noisy channel for data Observed Data Clean Source Data with Errors Noisy Channel Noisy Channel Model 1. We see an observation x in the noisy world w = arg max w ∈ W P ( w | x ) 2. Find the correct world w Applications: Speech, OCR, Spelling correction, Part of speech tagging, machine translations, etc…

Recommend

N 3 PDF Machine Learning PDFs QCD Introduction NISQ era We are in a Noisy

N 3 PDF Machine Learning PDFs QCD Introduction NISQ era We are in a Noisy

Towards Quantum Machine Learning Stefano Carrazza 19th October 2020, QTI TH meeting, CERN. Universit` a degli Studi di Milano, INFN Milan, CERN, TII N 3 PDF Machine Learning PDFs QCD Introduction NISQ era We are in a Noisy

604 views • 23 slides
Machine Learning for Healthcare 6.871, HST.956 Lecture 5: Learning with noisy or censored labels

Machine Learning for Healthcare 6.871, HST.956 Lecture 5: Learning with noisy or censored labels

Machine Learning for Healthcare 6.871, HST.956 Lecture 5: Learning with noisy or censored labels David Sontag Course announcements No recitation this Friday, but will be an extra office instead (2pm, 1-390) Problem set 1 due Mon Feb 24

419 views • 30 slides
A new perspective on machine learning H. N. Mhaskar Claremont Graduate University ICERM

A new perspective on machine learning H. N. Mhaskar Claremont Graduate University ICERM

A new perspective on machine learning H. N. Mhaskar Claremont Graduate University ICERM Scientific Machine Learning January 28, 2019 H. N. Mhaskar A new perspective on machine learning Outline My understanding of machine learning problem

657 views • 50 slides
Thermostatic Controls for Noisy Gradient Systems and Applications to Machine Learning Ben

Thermostatic Controls for Noisy Gradient Systems and Applications to Machine Learning Ben

Thermostatic Controls for Noisy Gradient Systems and Applications to Machine Learning Ben Leimkuhler University of Edinburgh Joint work with C. Matthews (Chicago), G. Stoltz (ENPC-Paris), M. Tretyakov (Nottingham) X. Shang (PhD student,

370 views • 34 slides
CSE217 INTRODUCTION TO DATA SCIENCE LECTURE 6: LEARNING PRINCIPLES Spring 2019 Marion Neumann

CSE217 INTRODUCTION TO DATA SCIENCE LECTURE 6: LEARNING PRINCIPLES Spring 2019 Marion Neumann

CSE217 INTRODUCTION TO DATA SCIENCE LECTURE 6: LEARNING PRINCIPLES Spring 2019 Marion Neumann RECAP: MACHINE LEARNING Workflow 2 NOISE noisy samples from true function 3 WHY IS NOISE A PROBLEM? small random sample from the noisy

746 views • 19 slides
Learning to denoise without clean data Joshua Batson hep-ai seminar 10/18/18 Noisy data is

Learning to denoise without clean data Joshua Batson hep-ai seminar 10/18/18 Noisy data is

Learning to denoise without clean data Joshua Batson hep-ai seminar 10/18/18 Noisy data is clean data + noise Noisy data is clean data + noise We want to predict You need a prior Prior: nearby pixels are similar Denoising strategy: local

546 views • 27 slides
Interactive Clustering Barna Saha Clustering Learning over Noisy Data Learn a classifier or

Interactive Clustering Barna Saha Clustering Learning over Noisy Data Learn a classifier or

Interactive Clustering Barna Saha Clustering Learning over Noisy Data Learn a classifier or find clusters over noisy/uncertain data Noise comes from using similarity func5onsadd an edge between two images if they represent the same

322 views • 13 slides
My Current Dream : Managing Machine Learning on Modern Hardware Ce Zhang 2 ML: fancy UDF

My Current Dream : Managing Machine Learning on Modern Hardware Ce Zhang 2 ML: fancy UDF

My Current Dream : Managing Machine Learning on Modern Hardware Ce Zhang 2 ML: fancy UDF breaking into traditional data workflow Input Recovered Ground Truth 1. Play well with existing data ecosystems (e.g., SciDB) 2. Fast! (< 20TB/s

519 views • 31 slides
Learning Explanatory Rules from Noisy Data Richard Evans, Ed Grefenstette Overview Our system,

Learning Explanatory Rules from Noisy Data Richard Evans, Ed Grefenstette Overview Our system,

Learning Explanatory Rules from Noisy Data Richard Evans, Ed Grefenstette Overview Our system, ILP, learns logic programs from examples. ILP learns by back-propagation. It is robust to noisy and ambiguous data. Learning Explanatory

638 views • 61 slides
Introduction to Machine Learning: Classification and The Noisy Channel Model CMSC 473/673 UMBC

Introduction to Machine Learning: Classification and The Noisy Channel Model CMSC 473/673 UMBC

Introduction to Machine Learning: Classification and The Noisy Channel Model CMSC 473/673 UMBC Some slides adapted from 3SLP Outline Classification Why incorporate uncertainty Classification with Bayes Rule Example: Email Classifier

1.68k views • 89 slides
Differential Privacy Machine Learning Li Xiong Big Data + Machine Learning + Machine

Differential Privacy Machine Learning Li Xiong Big Data + Machine Learning + Machine

CS573 Data Privacy and Security Differential Privacy Machine Learning Li Xiong Big Data + Machine Learning + Machine Learning Under Adversarial Settings Data privacy/confidentiality attacks membership attacks, model inversion

3.44k views • 63 slides
Introduction to machine learning COMS 4721 Learning from data Machine learning : the study

Introduction to machine learning COMS 4721 Learning from data Machine learning : the study

Introduction to machine learning COMS 4721 Learning from data Machine learning : the study of computational mechanisms that learn from data in order to make predictions and decisions. Example 1: image classification A birdwatcher

932 views • 14 slides
An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning http://www.cs.iastate.edu/~cs573x/bbsilab.html Machine Learning Software Preparing Data Building Classifiers Interpreting Results Machine Learning Software

496 views • 26 slides
Machine Learning Track Data Analytics, Machine Learning and HPC in todays changing

Machine Learning Track Data Analytics, Machine Learning and HPC in todays changing

Intel HPC Developer Convention Salt Lake City 2016 Machine Learning Track Data Analytics, Machine Learning and HPC in todays changing application environment Franz J. Kirly (practical) An overview of data analytics DATA Scientific

550 views • 27 slides
Better Machine Learning Through Data Sa Saleema ema Amershi shi Machine T eaching Group

Better Machine Learning Through Data Sa Saleema ema Amershi shi Machine T eaching Group

Better Machine Learning Through Data Sa Saleema ema Amershi shi Machine T eaching Group Microsoft Research August 14, 2016 Making better sense of data. Better data makes better machine learning. Data + Algorithm = Model Data Algorithm

1.16k views • 81 slides
x ? Machine Learning 5/4/20 Tim Althoff, UW CS547: Machine Learning for Big Data,

x ? Machine Learning 5/4/20 Tim Althoff, UW CS547: Machine Learning for Big Data,

? ? x ? Machine Learning 5/4/20 Tim Althoff, UW CS547: Machine Learning for Big Data, http://www.cs.washington.edu/cse547 2 ? ? ? ? Machine Learning ? Node classification 5/4/20 Tim Althoff, UW CS547: Machine Learning for Big Data,

755 views • 53 slides
Machine learning for machine data David Andrzejewski - @davidandrzej

Machine learning for machine data David Andrzejewski - @davidandrzej

Machine learning for machine data David Andrzejewski - @davidandrzej Data Sciences, Sumo Logic Strata Conference Machine Data Track February 13, 2014 1

1.1k views • 72 slides
ONLINE MACHINE LEARNING AND DATA MINING EDO LIBERTY STANDARD MACHINE LEARNING SETTING =

ONLINE MACHINE LEARNING AND DATA MINING EDO LIBERTY STANDARD MACHINE LEARNING SETTING =

ONLINE MACHINE LEARNING AND DATA MINING EDO LIBERTY STANDARD MACHINE LEARNING SETTING = call = ? = crawl Training Test Data Data Train call Apply Model Label 2 STANDARD MACHINE LEARNING SETTING Predicting the future

746 views • 51 slides
Machine Learning and Econometrics Hal Varian Jan 2014 Definitions Machine learning, data

Machine Learning and Econometrics Hal Varian Jan 2014 Definitions Machine learning, data

Machine Learning and Econometrics Hal Varian Jan 2014 Definitions Machine learning, data mining, predictive analytics, etc. all use data to predict some variable as a function of other variables. May or may not care about insight,

609 views • 44 slides
An inverse problem perspective on machine learning Lorenzo Rosasco University of Genova

An inverse problem perspective on machine learning Lorenzo Rosasco University of Genova

An inverse problem perspective on machine learning Lorenzo Rosasco University of Genova Massachusetts Institute of Technology Istituto Italiano di Tecnologia lcsl.mit.edu Feb 9th, 2018 Inverse Problems and Machine Learning Workshop, CM+X

765 views • 48 slides
A Probabilistic Model of Cross- situational Word Learning from Noisy and Ambiguous Data Afra

A Probabilistic Model of Cross- situational Word Learning from Noisy and Ambiguous Data Afra

A Probabilistic Model of Cross- situational Word Learning from Noisy and Ambiguous Data Afra Alishahi Joint work with Afsaneh Fazly and Suzanne Stevenson, University of Toronto 1 Word Learning Word learning: a mapping between a word and

885 views • 47 slides
Regression: Probabilistic perspective CE-717: Machine Learning Sharif University of Technology

Regression: Probabilistic perspective CE-717: Machine Learning Sharif University of Technology

Regression: Probabilistic perspective CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall 2018 Curve fitting: probabilistic perspective } Describing uncertainty over value of target variable as a probability distribution

438 views • 21 slides
Practical Advances in Machine Learning: A Computer Science Perspective Scott Neal Reilly &

Practical Advances in Machine Learning: A Computer Science Perspective Scott Neal Reilly &

Practical Advances in Machine Learning: A Computer Science Perspective Scott Neal Reilly & Jeff Druce Charles River Analytics Prepared for 2017 Workshop on Data Science and String Theory November 30 December 1, 2017 Objectives of this

1.13k views • 75 slides
Kernel Methods CE-717: Machine Learning Sharif University of Technology Fall 2019 Soleymani

Kernel Methods CE-717: Machine Learning Sharif University of Technology Fall 2019 Soleymani

Kernel Methods CE-717: Machine Learning Sharif University of Technology Fall 2019 Soleymani Not linearly separable data } Noisy data or overlapping classes 2 } (we discussed about it: soft margin) } Near linearly separable 1 }

677 views • 39 slides

More recommend