diagnose data for cleaning
play

Diagnose data for cleaning CLEAN IN G DATA IN P YTH ON Daniel - PowerPoint PPT Presentation

Jan 31, 2024 •386 likes •780 views

Diagnose data for cleaning CLEAN IN G DATA IN P YTH ON Daniel Chen Instructor Cleaning data Prepare data for analysis Data almost never comes in clean Diagnose your data for problems CLEANING DATA IN PYTHON Common data problems

  1. Diagnose data for cleaning CLEAN IN G DATA IN P YTH ON Daniel Chen Instructor

  2. Cleaning data Prepare data for analysis Data almost never comes in clean Diagnose your data for problems CLEANING DATA IN PYTHON

  3. Common data problems Inconsistent column names Missing data Outliers Duplicate rows Untidy Need to process columns Column types can signal unexpected data values CLEANING DATA IN PYTHON

  4. 1 2 3 4 5 Source: www.eea.europa.eu/data and maps/�gures/correlation between 6 7 8 fertility and female education CLEANING DATA IN PYTHON

  5. 1 2 3 4 5 Source: www.eea.europa.eu/data and maps/�gures/correlation between 6 7 8 fertility and female education CLEANING DATA IN PYTHON

  6. 1 2 3 4 5 Source: www.eea.europa.eu/data and maps/�gures/correlation between 6 7 8 fertility and female education CLEANING DATA IN PYTHON

  7. Column name inconsistencies 1 2 3 4 5 Source: www.eea.europa.eu/data and maps/�gures/correlation between 6 7 8 fertility and female education CLEANING DATA IN PYTHON

  8. Column name inconsistencies 1 2 3 4 5 Source: www.eea.europa.eu/data and maps/�gures/correlation between 6 7 8 fertility and female education CLEANING DATA IN PYTHON

  9. Column name inconsistencies 1 2 3 4 5 Source: www.eea.europa.eu/data and maps/�gures/correlation between 6 7 8 fertility and female education CLEANING DATA IN PYTHON

  10. Column name inconsistencies 1 2 3 4 5 Source: www.eea.europa.eu/data and maps/�gures/correlation between 6 7 8 fertility and female education CLEANING DATA IN PYTHON

  11. Column name inconsistencies Missing data 1 2 3 4 5 Source: www.eea.europa.eu/data and maps/�gures/correlation between 6 7 8 fertility and female education CLEANING DATA IN PYTHON

  12. Column name inconsistencies Missing data Country names are in French 1 2 3 4 5 Source: www.eea.europa.eu/data and maps/�gures/correlation between 6 7 8 fertility and female education CLEANING DATA IN PYTHON

  13. Load your data import pandas as pd df = pd.read_csv('literary_birth_rate.csv') CLEANING DATA IN PYTHON

  14. Visually inspect df.head() Continent Country female literacy fertility population 0 ASI Chine 90.5 1.769 1.324655e+09 1 ASI Inde 50.8 2.682 1.139965e+09 2 NAM USA 99.0 2.077 3.040600e+08 3 ASI Indonésie 88.8 2.132 2.273451e+08 4 LAT Brésil 90.2 1.827 NaN df.tail() Continent Country female literacy fertility population 0 AF Sao Tomé-et-Principe 90.5 1.769 1.324655e+09 1 LAT Aruba 50.8 2.682 1.139965e+09 2 ASI Tonga 99.0 2.077 3.040600e+08 3 OCE Australia 88.8 2.132 2.273451e+08 4 OCE Sweden 90.2 1.827 NaN CLEANING DATA IN PYTHON

  15. Visually inspect df.columns Index(['Continent', 'Country ', 'female literacy', 'fertility', 'population'], dtype='object') df.shape (164, 5) df.info() <class 'pandas.core.frame.DataFrame'> RangeIndex: 164 entries, 0 to 163 Data columns (total 5 columns): Continent 164 non-null object Country 164 non-null object female literacy 164 non-null float64 fertility 164 non-null object population 122 non-null float64 dtypes float64(2), object(3) memory usage: 6.5+ KB CLEANING DATA IN PYTHON

  16. Let's practice! CLEAN IN G DATA IN P YTH ON

  17. Exploratory data analysis CLEAN IN G DATA IN P YTH ON Daniel Chen Instructor

  18. Frequency counts Count the number of unique values in our data CLEANING DATA IN PYTHON

  19. Data type of each column df.info() <class 'pandas.core.frame.DataFrame'> RangeIndex: 164 entries, 0 to 163 Data columns (total 5 columns): continent 164 non-null object country 164 non-null object female literacy 164 non-null float64 fertility 164 non-null object population 122 non-null float64 dtypes float64(2), object(3) memory usage: 6.5+ KB CLEANING DATA IN PYTHON

  20. Frequency counts: continent df.continent.value_counts(dropna=False) AF 49 ASI 47 EUR 36 LAT 24 OCE 6 NAM 2 Name: continent, dtype: int64 CLEANING DATA IN PYTHON

  21. Frequency counts: continent df['continent'].value_counts(dropna=False) AF 49 ASI 47 EUR 36 LAT 24 OCE 6 NAM 2 Name: continent, dtype: int64 CLEANING DATA IN PYTHON

  22. Frequency counts: country df.country.value_counts(dropna=False).head() Sweden 2 Algerie 1 Germany 1 Angola 1 Indonésie 1 Name: country, dtype: int64 CLEANING DATA IN PYTHON

  23. Frequency counts: fertility df.fertility.value_counts(dropna=False).head() missing 5 1.854 2 1.93 2 1.841 2 1.393 2 Name: fertility, dtype: int64 CLEANING DATA IN PYTHON

  24. Frequency counts: population df.population.value_counts(dropna=False).head() NaN 42 5.667325e+06 1 3.773100e+06 1 1.333388e+06 1 1.661115e+08 1 Name: population, dtype: int64 CLEANING DATA IN PYTHON

  25. Summary statistics Numeric columns Outliers Considerably higher or lower Require further investigation CLEANING DATA IN PYTHON

  26. Summary statistics: numeric data df.describe() female_literacy population count 164.000000 1.220000e+02 mean 80.301220 6.345768e+07 std 22.977265 2.605977e+08 min 12.600000 1.035660e+05 25% 66.675000 3.778175e+06 50% 90.200000 9.995450e+06 75% 98.500000 2.642217e+07 max 100.000000 2.313000e+09 CLEANING DATA IN PYTHON

  27. Let's practice! CLEAN IN G DATA IN P YTH ON

  28. Visual exploratory data analysis CLEAN IN G DATA IN P YTH ON Daniel Chen Instructor

  29. Data visualization Great way to spot outliers and obvious errors More than just looking for patterns Plan data cleaning steps CLEANING DATA IN PYTHON

  30. Summary statistics df.describe() female_literacy fertility population count 164.000000 163.000000 1.220000e+02 mean 80.301220 2.872853 6.345768e+07 std 22.977265 1.425122 2.605977e+08 min 12.600000 0.966000 1.035660e+05 25% 66.675000 1.824500 3.778175e+06 50% 90.200000 2.362000 9.995450e+06 75% 98.500000 3.877500 2.642217e+07 max 100.000000 7.069000 2.313000e+09 CLEANING DATA IN PYTHON

  31. Bar plots and histograms Bar plots for discrete data counts Histograms for continuous data counts Look at frequencies CLEANING DATA IN PYTHON

  32. Histogram df.population.plot('hist') <matplotlib.axes._subplots.AxesSubplot at 0x7f78e4abafd0> import matplotlib.pyplot as plt plt.show() CLEANING DATA IN PYTHON

  33. Identifying the error df[df.population > 1000000000] continent country female literacy fertility population 0 ASI Chine 90.5 1.769 1.324655e+09 1 ASI Inde 50.8 2.682 1.139965e+09 162 OCE Australia 96.0 1.930 2.313000e+09 Not all outliers are bad data points Some can be an error, but others are valid values CLEANING DATA IN PYTHON

  34. Box plots Visualize basic summary statistics Outliers Min/max 25th, 50th, 75th percentiles CLEANING DATA IN PYTHON

  35. Box plot df.boxplot(column='population', by='continent') <matplotlib.axes._subplots.AxesSubplot at 0x7ff5581bb630> plt.show() CLEANING DATA IN PYTHON

  36. Box plot df.boxplot(column='population', by='continent') <matplotlib.axes._subplots.AxesSubplot at 0x7ff5581bb630> plt.show() CLEANING DATA IN PYTHON

  37. Scatter plots Relationship between 2 numeric variables Flag potentially bad data Errors not found by looking at 1 variable CLEANING DATA IN PYTHON

  38. Let's practice! CLEAN IN G DATA IN P YTH ON

Recommend

Diagnose data for cleaning Cleaning Data in Python Cleaning data Prepare data for analysis

Diagnose data for cleaning Cleaning Data in Python Cleaning data Prepare data for analysis

CLEANING DATA IN PYTHON Diagnose data for cleaning Cleaning Data in Python Cleaning data Prepare data for analysis Data almost never comes in clean Diagnose your data for problems Cleaning Data in Python Common data problems

1.08k views • 29 slides
What do we know? What do we know? (How do we diagnose?) (How do we diagnose?) Behavior

What do we know? What do we know? (How do we diagnose?) (How do we diagnose?) Behavior

Diagnosis and Misdiagnosis of Gifted Youth Presented at the 2012 California Psychological Association, Monterey, CA Daniel Peters, Ph.D. Stephen H. Chou, Psy.D. What do we know? What do we know? (How do we diagnose?) (How do we diagnose?)

172 views • 13 slides
Data cleaning and standardisation (1) Data cleaning and standardisation (2) Real world data is

Data cleaning and standardisation (1) Data cleaning and standardisation (2) Real world data is

Data cleaning and standardisation Febrl A parallel open source record linkage and geocoding system Record linkage and data integration Febrl overview Peter Christen Probabilistic data cleaning and standardisation Data Mining Group,

663 views • 5 slides
Workflow basics, RMarkdown, git/Github Cleaning up Cleaning up Cleaning up Cleaning up

Workflow basics, RMarkdown, git/Github Cleaning up Cleaning up Cleaning up Cleaning up

Workflow basics, RMarkdown, git/Github Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Cleaning up Don't worry, your history is preserved Settings Cleaning up Cleaning up Lots more options! RMarkdown Structure of an Rmd

656 views • 50 slides
Data Mining: A Powerful Data Mining: A Powerful Tool for Data Cleaning Tool for Data Cleaning

Data Mining: A Powerful Data Mining: A Powerful Tool for Data Cleaning Tool for Data Cleaning

Data Mining: A Powerful Data Mining: A Powerful Tool for Data Cleaning Tool for Data Cleaning Jiawei Han Department of Computer Science University of Illinois at Urbana-Champaign Nov. 4, 2003 1 Data mining for data quality assurance

721 views • 40 slides
Concatenating data Cleaning Data in Python Combining data Data may not always come in 1

Concatenating data Cleaning Data in Python Combining data Data may not always come in 1

CLEANING DATA IN PYTHON Concatenating data Cleaning Data in Python Combining data Data may not always come in 1 huge file 5 million row dataset may be broken into 5 separate datasets Easier to store and share May have

426 views • 26 slides
MORPH-II Dataset 1. Introduction to the Data 2. Inconsistencies in the Data 3. Cleaning the Data

MORPH-II Dataset 1. Introduction to the Data 2. Inconsistencies in the Data 3. Cleaning the Data

Garrett Bingham &amp; Ben Yip Summary and Cleaning June 16, 2017 University of North Carolina Wilmington MORPH-II Dataset 1. Introduction to the Data 2. Inconsistencies in the Data 3. Cleaning the Data 4. New Datasets 5. Dirty Data 6.

653 views • 26 slides
Intro to data cleaning with Apache Spark CLEAN IN G DATA W ITH P YS PARK Mike Metzger Data

Intro to data cleaning with Apache Spark CLEAN IN G DATA W ITH P YS PARK Mike Metzger Data

Intro to data cleaning with Apache Spark CLEAN IN G DATA W ITH P YS PARK Mike Metzger Data Engineering Consultant What is Data Cleaning? Data Cleaning : Preparing raw data for use in data processing pipelines. Possible tasks in data

1.03k views • 20 slides
Data Cleaning Nan Tang, QCRI Big Data Cleaning Nan Tang, QCRI Big Data Cleaning Nan Tang,

Data Cleaning Nan Tang, QCRI Big Data Cleaning Nan Tang, QCRI Big Data Cleaning Nan Tang,

Data Cleaning Nan Tang, QCRI Big Data Cleaning Nan Tang, QCRI Big Data Cleaning Nan Tang, QCRI Data Cleaning? 2 Data is Dirty 2 incomplete inconsistent inaccurate Data is Dirty 2 incomplete 25% companies: flawed data

1.07k views • 106 slides
Data types Cleaning Data in Python Prepare and clean data Cleaning Data in Python Data types

Data types Cleaning Data in Python Prepare and clean data Cleaning Data in Python Data types

CLEANING DATA IN PYTHON Data types Cleaning Data in Python Prepare and clean data Cleaning Data in Python Data types In [1]: print(df.dtypes) name object sex object treatment a object treatment b int64 dtype:

714 views • 44 slides
Data Preparation Data cleaning Data integration and transformation (Data

Data Preparation Data cleaning Data integration and transformation (Data

Data Preprocessing Why preprocess the data? Data Preparation Data cleaning Data integration and transformation (Data preprocessing) Data reduction, Feature selection Discretization and concept hierarchy generation 2

891 views • 21 slides
ASR Data Cleaning Guidelines Presenter: Asima Hameed Data Cleaning Process The speech files will

ASR Data Cleaning Guidelines Presenter: Asima Hameed Data Cleaning Process The speech files will

ASR Data Cleaning Guidelines Presenter: Asima Hameed Data Cleaning Process The speech files will be cleaned on the basis of: Silence Noise Mispronunciation Silence 1. A silence period of at least 200 ms 2. For voiced and voiceless

600 views • 26 slides
Outline Data cleaning and standardisation Febrl A parallel open source data linkage and

Outline Data cleaning and standardisation Febrl A parallel open source data linkage and

Outline Data cleaning and standardisation Febrl A parallel open source data linkage and geocoding system Data linkage and data integration Febrl overview Peter Christen, Tim Churches and others Probabilistic data cleaning and

833 views • 8 slides
DATA QUALITY AND DATA DATA QUALITY AND DATA PROGRAMMING PROGRAMMING &quot;Data cleaning and

DATA QUALITY AND DATA DATA QUALITY AND DATA PROGRAMMING PROGRAMMING &quot;Data cleaning and

DATA QUALITY AND DATA DATA QUALITY AND DATA PROGRAMMING PROGRAMMING &quot;Data cleaning and repairing account for about 60% of the work of data scientists.&quot; Christian Kaestner Required reading: Schelter, S., Lange, D., Schmidt, P.,

1.1k views • 86 slides
Data Handling: Import, Cleaning and Visualisation Lecture 7: Data Sources, Data Gathering, Data

Data Handling: Import, Cleaning and Visualisation Lecture 7: Data Sources, Data Gathering, Data

9/12/2019 Data Handling: Import, Cleaning and Visualisation Data Handling: Import, Cleaning and Visualisation Lecture 7: Data Sources, Data Gathering, Data Import Prof. Dr. Ulrich Matter 24/10/2019 file:///home/umatter/Dropbox/T

1.06k views • 54 slides
Data Preparation Data cleaning Discretization (Data preprocessing) Data

Data Preparation Data cleaning Discretization (Data preprocessing) Data

Data Preprocessing Why preprocess the data? Data Preparation Data cleaning Discretization (Data preprocessing) Data integration and transformation Data reduction, Feature selection 2 Why Prepare Data? Why Prepare

462 views • 15 slides
Data Handling: Import, Cleaning and Visualisation Lecture 3: Data Storage and Data Structures

Data Handling: Import, Cleaning and Visualisation Lecture 3: Data Storage and Data Structures

9/12/2019 Data Handling: Import, Cleaning and Visualisation Data Handling: Import, Cleaning and Visualisation Lecture 3: Data Storage and Data Structures Prof. Dr. Ulrich Matter 03/10/2019 file:///home/umatter/Dropbox/T

787 views • 62 slides
APPLYING BIG DATA ANALYTICS (BDA) TO DIAGNOSE HYDRO- METEOROLOGICAL RELATED RISK DUE TO CLIMATE

APPLYING BIG DATA ANALYTICS (BDA) TO DIAGNOSE HYDRO- METEOROLOGICAL RELATED RISK DUE TO CLIMATE

APPLYING BIG DATA ANALYTICS (BDA) TO DIAGNOSE HYDRO- METEOROLOGICAL RELATED RISK DUE TO CLIMATE CHANGE MOHD ZAKI M AMIN National Hydraulic Research Institute of Malaysia Ministry of Natural Resources &amp; Environment OCT . 19, 2016 OVERVIEW

739 views • 32 slides
Data Cleaning for Data Integration Advanced School on Data Exchange, Integration, and Streams

Data Cleaning for Data Integration Advanced School on Data Exchange, Integration, and Streams

Data Cleaning for Data Integration Advanced School on Data Exchange, Integration, and Streams (DEIS) Ekaterini Ioannou Tuesday, 9 th of Nov. 2010, Schloss Dagstuhl Problem overview Data integration: Combine data from various

806 views • 41 slides
Data Preparation Discretization Data cleaning (Data pre-processing) Data

Data Preparation Discretization Data cleaning (Data pre-processing) Data

Data Preparation Why prepare the data? Data Preparation Discretization Data cleaning (Data pre-processing) Data integration and transformation Data reduction, Feature selection 2 Why Prepare Data? Why Prepare Data?

426 views • 17 slides
Data Cleaning Mahdi Roozbahani Lecturer, Computational Science and Engineering, Georgia Tech

Data Cleaning Mahdi Roozbahani Lecturer, Computational Science and Engineering, Georgia Tech

Class Website CX4242: Data Cleaning Mahdi Roozbahani Lecturer, Computational Science and Engineering, Georgia Tech Data Cleaning How dirty is real data? How dirty is real data? Examples Jan 19, 2016 January 19, 16 1/19/16

309 views • 15 slides
Outline 0) Course Info 1) Introduction 2) Data Preparation and Cleaning 3) Schema matching and

Outline 0) Course Info 1) Introduction 2) Data Preparation and Cleaning 3) Schema matching and

Outline 0) Course Info 1) Introduction 2) Data Preparation and Cleaning 3) Schema matching and mapping 4) Virtual Data Integration 5) Data Exchange 6) Data Warehousing 7) Big Data Analytics 8) Data Provenance 1 CS520 - 1) Introduction Overview

1.63k views • 96 slides
Sequencing, data cleaning and assembling Swiss Institute of Bioinformatics (SIB) 26-30 November

Sequencing, data cleaning and assembling Swiss Institute of Bioinformatics (SIB) 26-30 November

Sequencing, data cleaning and assembling Swiss Institute of Bioinformatics (SIB) 26-30 November 2001 Course 2001 Sequencing, data cleaning and assembling. Some historical landmarks Organism Genome size (bp) Completed Bacteriophage X174

550 views • 32 slides
PhoneMD: Learning to Diagnose Parkinsons Disease from Smartphone Data Patrick Schwab and

PhoneMD: Learning to Diagnose Parkinsons Disease from Smartphone Data Patrick Schwab and

PhoneMD: Learning to Diagnose Parkinsons Disease from Smartphone Data Patrick Schwab and Walter Karlen @schwabpa @mhsl_ethz Institute of Robotics and Intelligent Systems ETH Zurich Parkinsons Disease (PD) Slow degeneration of motor

615 views • 50 slides

More recommend