data cleaning
play

Data Cleaning Mahdi Roozbahani Lecturer, Computational Science and - PowerPoint PPT Presentation

May 31, 2023 •142 likes •309 views

Class Website CX4242: Data Cleaning Mahdi Roozbahani Lecturer, Computational Science and Engineering, Georgia Tech Data Cleaning How dirty is real data? How dirty is real data? Examples Jan 19, 2016 January 19, 16 1/19/16

  1. Class Website CX4242: Data Cleaning Mahdi Roozbahani Lecturer, Computational Science and Engineering, Georgia Tech

  2. Data Cleaning How dirty is real data?

  3. How dirty is real data? Examples • Jan 19, 2016 • January 19, 16 • 1/19/16 • 2006-01-19 • 19/1/16 3 http://blogs.verdantis.com/wp-content/uploads/2015/02/Data-cleansing.jpg

  4. How dirty is real data? Discuss with you neighbors (group of 2-3) 60 seconds Comes up with 5+ kinds of “data dirtiness” 4

  5. How dirty is real data? • Missing or corrupted (NaN, null) • Numbers stored as string (“1232”) • Different units • Spelling/typos • Different string encodings • Outliers (due to data recording) • geocoding, timezone offsets (missing +, -) • Duplicate data • Fake data (malicious) • Sql injection • Different software version generating slightly different formats • Cap locks • Semi-colons • Structure (json objects) • Invisible characters • Different delimiters • Indentation 5

  6. Importance of Data Cleaning

  7. “80%” Time Spent on Data Preparation Cleaning Big Data: Most Time-Consuming, Least Enjoyable Data Science Task, Survey Says [Forbes] http://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time- consuming-least-enjoyable-data-science-task-survey-says/#73bf5b137f75 13

  8. Data Janitor

  9. Writing “Clean Code” • Be careful with trailing whitespaces • Indent code ( spaces vs tabs ) following coding practices in your team/company https://google.github.io/styleguide/javaguide.html#s4.2-block-indentation …there’s no way I'm going to be with someone who uses spaces over tabs… http://www.businessinsider.com/tabs-vs-spaces-from-silicon-valley-2016-5 Trailing whitespace is evil. Don't commit evil into your repo. http://codeimpossible.com/2012/04/02/trailing-whitespace-is-evil-don-t-commit-evil-into-your-repo/ 17

  10. Both available free for GT students on http://safaribooksonline.com/ 18

  11. Data Cleaners Watch videos • Data Wrangler (research at Stanford) • Open Refine (previously Google Refine ) Write down • Examples of data dirtiness • Tool’s features demo-ed (or that you like) Will collectively summarize similarities and differences afterwards Open Refine : http://openrefine.org Data Wrangler : http://vis.stanford.edu/wrangler/ 19

  14. What can Open Refine and Wrangler do? • [w,o] undo, redo • [o,w] history of data • [o] transform data (e.g., take log) • [w] data editing/highlighting/interaction may be easier • [o] clustering • [w] transpose/pivot • [w] fill in missing data • [w] suggestions + preview O = Open Refine W = Data wrangler 22

  15. ! The videos only show some of the tools’ features. Try them out. Open Refine : http://openrefine.org Data Wrangler : http://vis.stanford.edu/wrangler/ 37

Recommend

Data cleaning and standardisation (1) Data cleaning and standardisation (2) Real world data is

Data cleaning and standardisation (1) Data cleaning and standardisation (2) Real world data is

Data cleaning and standardisation Febrl A parallel open source record linkage and geocoding system Record linkage and data integration Febrl overview Peter Christen Probabilistic data cleaning and standardisation Data Mining Group,

663 views • 5 slides
Workflow basics, RMarkdown, git/Github Cleaning up Cleaning up Cleaning up Cleaning up

Workflow basics, RMarkdown, git/Github Cleaning up Cleaning up Cleaning up Cleaning up

Workflow basics, RMarkdown, git/Github Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Don't worry, your history is preserved Settings Cleaning up Cleaning up Lots more options! RMarkdown Structure of an Rmd

656 views • 50 slides
Data Mining: A Powerful Data Mining: A Powerful Tool for Data Cleaning Tool for Data Cleaning

Data Mining: A Powerful Data Mining: A Powerful Tool for Data Cleaning Tool for Data Cleaning

Data Mining: A Powerful Data Mining: A Powerful Tool for Data Cleaning Tool for Data Cleaning Jiawei Han Department of Computer Science University of Illinois at Urbana-Champaign Nov. 4, 2003 1 Data mining for data quality assurance

721 views • 40 slides
Concatenating data Cleaning Data in Python Combining data Data may not always come in 1

Concatenating data Cleaning Data in Python Combining data Data may not always come in 1

CLEANING DATA IN PYTHON Concatenating data Cleaning Data in Python Combining data Data may not always come in 1 huge file 5 million row dataset may be broken into 5 separate datasets Easier to store and share May have

425 views • 26 slides
MORPH-II Dataset 1. Introduction to the Data 2. Inconsistencies in the Data 3. Cleaning the Data

MORPH-II Dataset 1. Introduction to the Data 2. Inconsistencies in the Data 3. Cleaning the Data

Garrett Bingham & Ben Yip Summary and Cleaning June 16, 2017 University of North Carolina Wilmington MORPH-II Dataset 1. Introduction to the Data 2. Inconsistencies in the Data 3. Cleaning the Data 4. New Datasets 5. Dirty Data 6.

653 views • 26 slides
Intro to data cleaning with Apache Spark CLEAN IN G DATA W ITH P YS PARK Mike Metzger Data

Intro to data cleaning with Apache Spark CLEAN IN G DATA W ITH P YS PARK Mike Metzger Data

Intro to data cleaning with Apache Spark CLEAN IN G DATA W ITH P YS PARK Mike Metzger Data Engineering Consultant What is Data Cleaning? Data Cleaning : Preparing raw data for use in data processing pipelines. Possible tasks in data

1.03k views • 20 slides
Diagnose data for cleaning Cleaning Data in Python Cleaning data Prepare data for analysis

Diagnose data for cleaning Cleaning Data in Python Cleaning data Prepare data for analysis

CLEANING DATA IN PYTHON Diagnose data for cleaning Cleaning Data in Python Cleaning data Prepare data for analysis Data almost never comes in clean Diagnose your data for problems Cleaning Data in Python Common data problems

1.08k views • 29 slides
Data Cleaning Nan Tang, QCRI Big Data Cleaning Nan Tang, QCRI Big Data Cleaning Nan Tang,

Data Cleaning Nan Tang, QCRI Big Data Cleaning Nan Tang, QCRI Big Data Cleaning Nan Tang,

Data Cleaning Nan Tang, QCRI Big Data Cleaning Nan Tang, QCRI Big Data Cleaning Nan Tang, QCRI Data Cleaning? 2 Data is Dirty 2 incomplete inconsistent inaccurate Data is Dirty 2 incomplete 25% companies: flawed data

1.07k views • 106 slides
Diagnose data for cleaning CLEAN IN G DATA IN P YTH ON Daniel Chen Instructor Cleaning data

Diagnose data for cleaning CLEAN IN G DATA IN P YTH ON Daniel Chen Instructor Cleaning data

Diagnose data for cleaning CLEAN IN G DATA IN P YTH ON Daniel Chen Instructor Cleaning data Prepare data for analysis Data almost never comes in clean Diagnose your data for problems CLEANING DATA IN PYTHON Common data problems

780 views • 38 slides
Data types Cleaning Data in Python Prepare and clean data Cleaning Data in Python Data types

Data types Cleaning Data in Python Prepare and clean data Cleaning Data in Python Data types

CLEANING DATA IN PYTHON Data types Cleaning Data in Python Prepare and clean data Cleaning Data in Python Data types In [1]: print(df.dtypes) name object sex object treatment a object treatment b int64 dtype:

714 views • 44 slides
Data Preparation Data cleaning Data integration and transformation (Data

Data Preparation Data cleaning Data integration and transformation (Data

Data Preprocessing Why preprocess the data? Data Preparation Data cleaning Data integration and transformation (Data preprocessing) Data reduction, Feature selection Discretization and concept hierarchy generation 2

891 views • 21 slides
ASR Data Cleaning Guidelines Presenter: Asima Hameed Data Cleaning Process The speech files will

ASR Data Cleaning Guidelines Presenter: Asima Hameed Data Cleaning Process The speech files will

ASR Data Cleaning Guidelines Presenter: Asima Hameed Data Cleaning Process The speech files will be cleaned on the basis of: Silence Noise Mispronunciation Silence 1. A silence period of at least 200 ms 2. For voiced and voiceless

600 views • 26 slides
Outline Data cleaning and standardisation Febrl A parallel open source data linkage and

Outline Data cleaning and standardisation Febrl A parallel open source data linkage and

Outline Data cleaning and standardisation Febrl A parallel open source data linkage and geocoding system Data linkage and data integration Febrl overview Peter Christen, Tim Churches and others Probabilistic data cleaning and

833 views • 8 slides
DATA QUALITY AND DATA DATA QUALITY AND DATA PROGRAMMING PROGRAMMING "Data cleaning and

DATA QUALITY AND DATA DATA QUALITY AND DATA PROGRAMMING PROGRAMMING "Data cleaning and

DATA QUALITY AND DATA DATA QUALITY AND DATA PROGRAMMING PROGRAMMING "Data cleaning and repairing account for about 60% of the work of data scientists." Christian Kaestner Required reading: Schelter, S., Lange, D., Schmidt, P.,

1.1k views • 86 slides
Data Handling: Import, Cleaning and Visualisation Lecture 7: Data Sources, Data Gathering, Data

Data Handling: Import, Cleaning and Visualisation Lecture 7: Data Sources, Data Gathering, Data

9/12/2019 Data Handling: Import, Cleaning and Visualisation Data Handling: Import, Cleaning and Visualisation Lecture 7: Data Sources, Data Gathering, Data Import Prof. Dr. Ulrich Matter 24/10/2019 file:///home/umatter/Dropbox/T

1.06k views • 54 slides
Data Preparation Data cleaning Discretization (Data preprocessing) Data

Data Preparation Data cleaning Discretization (Data preprocessing) Data

Data Preprocessing Why preprocess the data? Data Preparation Data cleaning Discretization (Data preprocessing) Data integration and transformation Data reduction, Feature selection 2 Why Prepare Data? Why Prepare

462 views • 15 slides
Data Handling: Import, Cleaning and Visualisation Lecture 3: Data Storage and Data Structures

Data Handling: Import, Cleaning and Visualisation Lecture 3: Data Storage and Data Structures

9/12/2019 Data Handling: Import, Cleaning and Visualisation Data Handling: Import, Cleaning and Visualisation Lecture 3: Data Storage and Data Structures Prof. Dr. Ulrich Matter 03/10/2019 file:///home/umatter/Dropbox/T

787 views • 62 slides
Data Cleaning for Data Integration Advanced School on Data Exchange, Integration, and Streams

Data Cleaning for Data Integration Advanced School on Data Exchange, Integration, and Streams

Data Cleaning for Data Integration Advanced School on Data Exchange, Integration, and Streams (DEIS) Ekaterini Ioannou Tuesday, 9 th of Nov. 2010, Schloss Dagstuhl Problem overview Data integration: Combine data from various

806 views • 41 slides
Data Preparation Discretization Data cleaning (Data pre-processing) Data

Data Preparation Discretization Data cleaning (Data pre-processing) Data

Data Preparation Why prepare the data? Data Preparation Discretization Data cleaning (Data pre-processing) Data integration and transformation Data reduction, Feature selection 2 Why Prepare Data? Why Prepare Data?

426 views • 17 slides
Outline 0) Course Info 1) Introduction 2) Data Preparation and Cleaning 3) Schema matching and

Outline 0) Course Info 1) Introduction 2) Data Preparation and Cleaning 3) Schema matching and

Outline 0) Course Info 1) Introduction 2) Data Preparation and Cleaning 3) Schema matching and mapping 4) Virtual Data Integration 5) Data Exchange 6) Data Warehousing 7) Big Data Analytics 8) Data Provenance 1 CS520 - 1) Introduction Overview

1.63k views • 96 slides
Sequencing, data cleaning and assembling Swiss Institute of Bioinformatics (SIB) 26-30 November

Sequencing, data cleaning and assembling Swiss Institute of Bioinformatics (SIB) 26-30 November

Sequencing, data cleaning and assembling Swiss Institute of Bioinformatics (SIB) 26-30 November 2001 Course 2001 Sequencing, data cleaning and assembling. Some historical landmarks Organism Genome size (bp) Completed Bacteriophage X174

550 views • 32 slides
Data Cleaning Mahdi Roozbahani Lecturer, Computational Science and Engineering, Georgia Tech

Data Cleaning Mahdi Roozbahani Lecturer, Computational Science and Engineering, Georgia Tech

CX4242: Data Cleaning Mahdi Roozbahani Lecturer, Computational Science and Engineering, Georgia Tech Data Cleaning How dirty is real data? How dirty is real data? Examples Jan 19, 2016 January 19, 16 1/19/16 2006-01-19

763 views • 19 slides
Outline 0) Course Info 1) Introduction 2) Data Preparation and Cleaning 3) Schema mappings and

Outline 0) Course Info 1) Introduction 2) Data Preparation and Cleaning 3) Schema mappings and

Outline 0) Course Info 1) Introduction 2) Data Preparation and Cleaning 3) Schema mappings and Virtual Data Integration 4) Data Exchange 5) Data Warehousing 6) Big Data Analytics 7) Data Provenance 1 CS520 - Course Info About me I am a database

781 views • 34 slides
Outline 0) Course Info 1) Introduction 2) Data Preparation and Cleaning 3) Schema matching and

Outline 0) Course Info 1) Introduction 2) Data Preparation and Cleaning 3) Schema matching and

Outline 0) Course Info 1) Introduction 2) Data Preparation and Cleaning 3) Schema matching and mapping 4) Virtual Data Integration 5) Data Exchange 6) Data Warehousing 7) Big Data Analytics 8) Data Provenance 1 CS520 - 1) Introduction 2.

1.02k views • 87 slides

More recommend