mean median mode imputations
play

Mean, median & mode imputations DEALIN G W ITH MIS S IN G DATA - PowerPoint PPT Presentation

Nov 13, 2023 •406 likes •821 views

Mean, median & mode imputations DEALIN G W ITH MIS S IN G DATA IN P YTH ON Suraj Donthi Deep Learning & Computer Vision Consultant Basic imputation techniques constant (e.g. 0) mean median mode or most frequent DEALING WITH

  1. Mean, median & mode imputations DEALIN G W ITH MIS S IN G DATA IN P YTH ON Suraj Donthi Deep Learning & Computer Vision Consultant

  2. Basic imputation techniques constant (e.g. 0) mean median mode or most frequent DEALING WITH MISSING DATA IN PYTHON

  3. Mean Imputation from sklearn.impute import SimpleImputer diabetes_mean = diabetes.copy(deep=True) mean_imputer = SimpleImputer(strategy='mean') DEALING WITH MISSING DATA IN PYTHON

  4. Mean Imputation from sklearn.impute import SimpleImputer diabetes_mean = diabetes.copy(deep=True) mean_imputer = SimpleImputer(strategy='mean') diabetes_mean.iloc[:, :] = mean_imputer.fit_transform(diabetes_mean) DEALING WITH MISSING DATA IN PYTHON

  5. Median imputation diabetes_median = diabetes.copy(deep=True) median_imputer = SimpleImputer(strategy='median') diabetes_median.iloc[:, :] = median_imputer.fit_transform(diabetes_median) DEALING WITH MISSING DATA IN PYTHON

  6. Mode imputation diabetes_mode = diabetes.copy(deep=True) mode_imputer = SimpleImputer(strategy='most_frequent') diabetes_mode.iloc[:, :] = mode_imputer.fit_transform(diabetes_mode) DEALING WITH MISSING DATA IN PYTHON

  7. Imputing a constant diabetes_constant = diabetes.copy(deep=True) constant_imputer = SimpleImputer(strategy='constant', fill_value=0)) diabetes_constant.iloc[:, :] = constant_imputer.fit_transform(diabetes_constant) DEALING WITH MISSING DATA IN PYTHON

  8. Scatterplot of imputation nullity = diabetes['Serum_Insulin'].isnull()+diabetes['Glucose'].isnull() diabetes_mean.plot(x='Serum_Insulin', y='Glucose', kind='scatter', alpha=0.5, c=nullity, cmap='rainbow', title='Mean Imputation') DEALING WITH MISSING DATA IN PYTHON

  9. Visualizing imputations fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(10, 10)) nullity = diabetes['Serum_Insulin'].isnull()+diabetes['Glucose'].isnull() imputations = {'Mean Imputation': diabetes_mean, 'Median Imputation': diabetes_median, 'Most Frequent Imputation': diabetes_mode, 'Constant Imputation': diabetes_constant} for ax, df_key in zip(axes.flatten(), imputations): imputations[df_key].plot(x='Serum_Insulin', y='Glucose', kind='scatter', alpha=0.5, c=nullity, cmap='rainbow', ax=ax, colorbar=False, title=df_key) DEALING WITH MISSING DATA IN PYTHON

  10. DEALING WITH MISSING DATA IN PYTHON

  11. Summary You learned to Impute with statistical parameters like mean, median and mode Graphically compare the imputations Analyze the imputations DEALING WITH MISSING DATA IN PYTHON

  12. Let's practice! DEALIN G W ITH MIS S IN G DATA IN P YTH ON

  13. Imputing time-series data DEALIN G W ITH MIS S IN G DATA IN P YTH ON Suraj Donthi Deep Learning & Computer Vision Consultant

  14. Airquality Dataset import pandas as pd airquality = pd.read_csv('air-quality.csv', parse_dates='Date', index_col='Date') airquality.head() Ozone Solar Wind Temp Date 1976-05-01 41.0 190.0 7.4 67 1976-05-02 36.0 118.0 8.0 72 1976-05-03 12.0 149.0 12.6 74 1976-05-04 18.0 313.0 11.5 62 1976-05-05 NaN NaN 14.3 56 DEALING WITH MISSING DATA IN PYTHON

  15. Airquality Dataset airquality.isnull().sum() airquality.isnull.mean() * 100 Ozone 37 Ozone 24.183007 Solar 7 Solar 4.575163 Wind 0 Wind 0.000000 Temp 0 Temp 0.000000 dtype: int64 dtype: float64 DEALING WITH MISSING DATA IN PYTHON

  16. The .�llna() method The attribute method in .fillna() can be set to 'ffill' or 'pad' 'bfill' or 'backwardfill' DEALING WITH MISSING DATA IN PYTHON

  17. F�ll method Replace NaN s with last observed value pad is the same as 'ffill' airquality.fillna(method='ffill', inplace=True) DEALING WITH MISSING DATA IN PYTHON

  18. airquality.fillna(method='ffill', inplace=True) airquality['Ozone'][30:40] airquality['Ozone'][30:40] Date Ozone Date Ozone 1976-05-31 37.0 1976-05-31 37.0 1976-06-01 NaN 1976-06-01 37.0 1976-06-02 NaN 1976-06-02 37.0 1976-06-03 NaN 1976-06-03 37.0 1976-06-04 NaN 1976-06-04 37.0 1976-06-05 NaN 1976-06-05 37.0 1976-06-06 NaN 1976-06-06 37.0 1976-06-07 29.0 1976-06-07 29.0 1976-06-08 NaN 1976-06-08 29.0 1976-06-09 71.0 1976-06-09 71.0 DEALING WITH MISSING DATA IN PYTHON

  19. B�ll method Replace NaN s with next observed value backfill is the same as 'bfill' df.fillna(method='bfill', inplace=True) DEALING WITH MISSING DATA IN PYTHON

  20. airquality.fillna(method='bfill', inplace=True) airquality['Ozone'][30:40] airquality['Ozone'][30:40] Date Ozone Date Ozone 1976-05-31 37.0 1976-05-31 37.0 1976-06-01 NaN 1976-06-01 29.0 1976-06-02 NaN 1976-06-02 29.0 1976-06-03 NaN 1976-06-03 29.0 1976-06-04 NaN 1976-06-04 29.0 1976-06-05 NaN 1976-06-05 29.0 1976-06-06 NaN 1976-06-06 29.0 1976-06-07 29.0 1976-06-07 29.0 1976-06-08 NaN 1976-06-08 71.0 1976-06-09 71.0 1976-06-09 71.0 DEALING WITH MISSING DATA IN PYTHON

  21. The .interpolate() method The .interpolate() method extends the sequence of values to the missing values The attribute method in .interpolate() can be set to 'linear' 'quadratic' 'nearest' DEALING WITH MISSING DATA IN PYTHON

  22. Linear interpolation Impute linearly or with equidistant values df.interpolate(method='linear', inplace=True) DEALING WITH MISSING DATA IN PYTHON

  23. airquality.interpolate( method='linear', inplace=True) airquality['Ozone'][30:40] airquality['Ozone'][30:40] Date Ozone Date Ozone 1976-05-31 37.0 1976-05-31 37.0 1976-06-01 NaN 1976-06-01 35.9 1976-06-02 NaN 1976-06-02 34.7 1976-06-03 NaN 1976-06-03 33.6 1976-06-04 NaN 1976-06-04 32.4 1976-06-05 NaN 1976-06-05 31.3 1976-06-06 NaN 1976-06-06 30.1 1976-06-07 29.0 1976-06-07 29.0 1976-06-08 NaN 1976-06-08 50.0 1976-06-09 71.0 1976-06-09 71.0 DEALING WITH MISSING DATA IN PYTHON

  24. Quadratic interpolation Impute the values quadratically df.interpolate(method='quadratic', inplace=True) DEALING WITH MISSING DATA IN PYTHON

  25. airquality.interpolate( method='quadratic', inplace=True) airquality['Ozone'][30:39] airquality['Ozone'][30:39] Ozone Ozone Date Date 1976-05-31 37.0 1976-05-31 37.0 1976-06-01 NaN 1976-06-01 -38.4 1976-06-02 NaN 1976-06-02 -79.4 1976-06-03 NaN 1976-06-03 -85.9 1976-06-04 NaN 1976-06-04 -62.4 1976-06-05 NaN 1976-06-06 -2.8 1976-06-06 NaN 1976-06-07 29.0 1976-06-07 29.0 1976-06-08 62.2 1976-06-08 NaN DEALING WITH MISSING DATA IN PYTHON

  26. Nearest value imputation Impute with the nearest observable value df.interpolate(method='nearest', inplace=True) DEALING WITH MISSING DATA IN PYTHON

  27. airquality.interpolate( method='nearest', inplace=True) airquality['Ozone'][30:39] airquality['Ozone'][30:39] Date Ozone Date Ozone 1976-05-31 37.0 1976-05-31 37.0 1976-06-01 NaN 1976-06-01 37.0 1976-06-02 NaN 1976-06-02 37.0 1976-06-03 NaN 1976-06-03 37.0 1976-06-04 NaN 1976-06-04 29.0 1976-06-05 NaN 1976-06-05 29.0 1976-06-06 NaN 1976-06-06 29.0 1976-06-07 29.0 1976-06-07 29.0 1976-06-08 NaN 1976-06-08 29.0 DEALING WITH MISSING DATA IN PYTHON

  28. Let's practice! DEALIN G W ITH MIS S IN G DATA IN P YTH ON

  29. Visualizing time- series imputations DEALIN G W ITH MIS S IN G DATA IN P YTH ON Suraj Donthi Deep Learning & Computer Vision Learning

  30. Air quality time-series plot airquality['Ozone'].plot(title='Ozone', marker='o', figsize=(30, 5)) DEALING WITH MISSING DATA IN PYTHON

  31. F�ll Imputation ffill_imp['Ozone'].plot(color='red', marker='o', linestyle='dotted', figsize=(30, 5)) airquality['Ozone'].plot(title='Ozone', marker='o') DEALING WITH MISSING DATA IN PYTHON

  32. B�ll Imputation bfill_imp['Ozone'].plot(color='red', marker='o', linestyle='dotted', figsize=(30, 5)) airquality['Ozone'].plot(title='Ozone', marker='o') DEALING WITH MISSING DATA IN PYTHON

  33. Linear Interpolation linear_interp['Ozone'].plot(color='red', marker='o', linestyle='dotted', figsize=(30, 5) airquality['Ozone'].plot(title='Ozone', marker='o') DEALING WITH MISSING DATA IN PYTHON

  34. Quadratic Interpolation quadratic_interp['Ozone'].plot(color='red', marker='o', linestyle='dotted', figsize=(30, 5)) airquality['Ozone'].plot(title='Ozone', marker='o') DEALING WITH MISSING DATA IN PYTHON

  35. Nearest Interpolation nearest_interp['Ozone'].plot(color='red', marker='o', linestyle='dotted', figsize=(30, 5 airquality['Ozone'].plot(title='Ozone', marker='o') DEALING WITH MISSING DATA IN PYTHON

  36. A comparison of the interpolations # Create subplots fig, axes = plt.subplots(3, 1, figsize=(30, 20)) # Create interpolations dictionary interpolations = {'Linear Interpolation': linear_interp, 'Quadratic Interpolation': quadratic_interp, 'Nearest Interpolation': nearest_interp} # Visualize each interpolation for ax, df_key in zip(axes, interpolations): interpolations[df_key].Ozone.plot(color='red', marker='o', linestyle='dotted', ax=ax) airquality.Ozone.plot(title=df_key + ' - Ozone', marker='o', ax=ax) DEALING WITH MISSING DATA IN PYTHON

  37. A comparison of the interpolations DEALING WITH MISSING DATA IN PYTHON

  38. A comparison of imputation techniques DEALING WITH MISSING DATA IN PYTHON

Recommend

the nerves sensory radial median ulnar median median sensory median median ulnar radial

the nerves sensory radial median ulnar median median sensory median median ulnar radial

the nerves sensory radial median ulnar median median sensory median median ulnar radial median ulnar http://www.eatonhand.com/jpg/1494401.jpg Image Credit: Donnie Ray Jones on Flickr | CC License |

618 views • 17 slides
Sensitivity of the population size estimates for different imputations of a covariate B.F.M.

Sensitivity of the population size estimates for different imputations of a covariate B.F.M.

C APTURE -R ECAPTURE U SUAL R ESIDENTS S CENARIO S E XPECTATIONS R ESULTS Sensitivity of the population size estimates for different imputations of a covariate B.F.M. Bakker 2 P.G.M. van der Heijden 3 S.C. Gerritse Utrecht University 2

557 views • 22 slides
Numerical summary of data Measures of location: mode , median, mean, Measures of spread:

Numerical summary of data Measures of location: mode , median, mean, Measures of spread:

Introduction to Statistics Numerical summary of data Measures of location: mode , median, mean, Measures of spread: range, interquartile range, standard deviation , Measures of form: skewness, kurtosis , Introduction to

614 views • 30 slides
Using the Superpopulation Model for Imputations and Variance Computation in Survey Sampling Petr

Using the Superpopulation Model for Imputations and Variance Computation in Survey Sampling Petr

Using the Superpopulation Model for Imputations and Variance Computation in Survey Sampling Petr Novk, Vclav Kosina Czech Statistical Office Petr Novk, Vclav Kosina Using the Superpopulation Model for Imputations and Variance

355 views • 19 slides
Assumptions and normal distributions EX P ERIMEN TAL DES IGN IN P YTH ON Luke Hayden

Assumptions and normal distributions EX P ERIMEN TAL DES IGN IN P YTH ON Luke Hayden

Assumptions and normal distributions EX P ERIMEN TAL DES IGN IN P YTH ON Luke Hayden Instructor Summary stats Mean Sum divided by count Median Half of values fall above and below the median Mode Value that occurs most often Standard

745 views • 41 slides
Linear-time Median Def: Median of elements A=a 1 , a 2 , , a n is the (n/2)-th smallest element

Linear-time Median Def: Median of elements A=a 1 , a 2 , , a n is the (n/2)-th smallest element

Linear-time Median Def: Median of elements A=a 1 , a 2 , , a n is the (n/2)-th smallest element in A. How to find median? sort the elements, output the elem. at (n/2)-th position - running time ? Linear-time Median Def: Median of

229 views • 11 slides
African American Strategy Equitable Access to Homeownership Presentation April 16, 2018

African American Strategy Equitable Access to Homeownership Presentation April 16, 2018

CON FI DEN T I AL African American Strategy Equitable Access to Homeownership Presentation April 16, 2018 Individual Demographics Age Income (1000s) FICO Total DTI All Debt Student Debt ($) Median Median Median Median Median Median

299 views • 13 slides
Median Finding Test Cases What's Next 1. Median finding, part 2 2. Why we write test cases 3.

Median Finding Test Cases What's Next 1. Median finding, part 2 2. Why we write test cases 3.

Median Finding Test Cases What's Next 1. Median finding, part 2 2. Why we write test cases 3. What's next? A problem Find the (upper) median of a list of n items. Upper median means "if the list has an even number of items, pick

1.38k views • 52 slides
Median filter Non-linear filtering example Replace each pixel by the median over N pixels (5

Median filter Non-linear filtering example Replace each pixel by the median over N pixels (5

Today Median filter Non-linear filtering example Replace each pixel by the median over N pixels (5 pixels, for these examples). Generalizes to rank order filters. Spike In: Out: noise is removed 5-pixel neighborhood Monotonic

432 views • 16 slides
Events MedIAN Jobs Contact The Network Background About MedIAN UKs national Medical

Events MedIAN Jobs Contact The Network Background About MedIAN UKs national Medical

About Events MedIAN Jobs Contact The Network Background About MedIAN UKs national Medical Image Analysis community A network of academics , clinicians , and industry Steering New clinical research and tech collaborations

728 views • 15 slides
Business Statistics CONTENTS Hypotheses on the median The sign test The Wilcoxon signed ranks

Business Statistics CONTENTS Hypotheses on the median The sign test The Wilcoxon signed ranks

MEDIAN: NON-PARAMETRIC TESTS Business Statistics CONTENTS Hypotheses on the median The sign test The Wilcoxon signed ranks test Old exam question Further study HYPOTHESES ON THE MEDIAN The median is a central value that may be more

327 views • 19 slides
Today Non-linear filtering example Median filter Replace each pixel by the median over N pixels

Today Non-linear filtering example Median filter Replace each pixel by the median over N pixels

Today Non-linear filtering example Median filter Replace each pixel by the median over N pixels (5 pixels, for these examples). Generalizes to rank order filters. Spike In: Out: noise is removed 5-pixel neighborhood Monotonic

1.25k views • 91 slides
City of Atascadero Size: 26 sq miles Incorporated: 1979 Population: 30,305 o Median

City of Atascadero Size: 26 sq miles Incorporated: 1979 Population: 30,305 o Median

City of Atascadero Size: 26 sq miles Incorporated: 1979 Population: 30,305 o Median Age 38.2 yrs (US Median 37.8 yrs ) o Avg. Family Income $104,744 (County $104,489) o Total # of Households 11,431 Median Home Price: $480,000 City

567 views • 14 slides
W4231: Analysis of Algorithms Definition of median 9/14/1999 Let A = a 1 a n be a

W4231: Analysis of Algorithms Definition of median 9/14/1999 Let A = a 1 a n be a

W4231: Analysis of Algorithms Definition of median 9/14/1999 Let A = a 1 a n be a sequence of integers. The median of A is a value v such that Median Selection |{ i : a i < v }| n/ 2 and |{ i : a i > v }| n/ 2 That is,

218 views • 4 slides
Harbor Bay Median Sheet Mulch project HARBOR BAY MEDIAN Located in the City of Alameda 2

Harbor Bay Median Sheet Mulch project HARBOR BAY MEDIAN Located in the City of Alameda 2

Short case study of Harbor Bay Median Sheet Mulch project HARBOR BAY MEDIAN Located in the City of Alameda 2 miles of medians along a busy road Project located in environmentally sensitive area less than 100 yards from shoreline

392 views • 11 slides
Median Filter (5) sparsely populated 5x5 median filters effect on white noise; std of output

Median Filter (5) sparsely populated 5x5 median filters effect on white noise; std of output

Castleman, 1996 Median Filter (5) sparsely populated 5x5 median filters effect on white noise; std of output image vs. # pixels participating 41 IIP Sharpening (1) Used to: highlight fine detail Enhance detail that has been

305 views • 8 slides
Org-mode Nick Higham April 22, 2013 Nick Higham Org-mode 1 / 7 University of Manchester What

Org-mode Nick Higham April 22, 2013 Nick Higham Org-mode 1 / 7 University of Manchester What

Org-mode Nick Higham April 22, 2013 Nick Higham Org-mode 1 / 7 University of Manchester What is Org Mode? Emacs mode for note taking task management tables and spreadsheets producing L A T EX documents and web pages literate

212 views • 7 slides
I - -75 Median Cable Barrier 75 Median Cable Barrier 75 Median Cable Barrier I 75 Median Cable

I - -75 Median Cable Barrier 75 Median Cable Barrier 75 Median Cable Barrier I 75 Median Cable

I - -75 Median Cable Barrier 75 Median Cable Barrier 75 Median Cable Barrier I 75 Median Cable Barrier Design Build Project Design Build Project Design Build Project Design Build Project Florida Department Of Transportation District One

508 views • 16 slides
Lee @ RobinsonSIR.com 513.842.2225 2 3 4 5 6

Lee @ RobinsonSIR.com 513.842.2225 2 3 4 5 6

Lee @ RobinsonSIR.com 513.842.2225 2 3 4 5 6 7 8 9 The median price in the Midwest declined to $141,700 in August 2011 from a median $146,300 in July 2011. The median price also fell 3.5%

252 views • 22 slides
Educational topic: FT8 digital mode What is FT8 Digital operating mode that started gaining

Educational topic: FT8 digital mode What is FT8 Digital operating mode that started gaining

Educational topic: FT8 digital mode What is FT8 Digital operating mode that started gaining momentum in 2017 Named after Its developers Steven Franke (K9AN) Joe Taylor (K1JT) Mode uses an 8-frequency shift keying format Signals occupy 50 Hz

472 views • 21 slides
Spartanburg Nation Median Value of a $115,900 $184,700 Home Median Gross Rent $705 $950

Spartanburg Nation Median Value of a $115,900 $184,700 Home Median Gross Rent $705 $950

Spartanburg Nation Median Value of a $115,900 $184,700 Home Median Gross Rent $705 $950 per Month Spartanburg Nation Median Value of a $115,900 $184,700 Home Median Gross Rent $705 $950 per Month Median Household $35,650

378 views • 26 slides
Median Incomes and Rent Growth Here We Go Again Bud Clarke, MAI Director, MMA Investment

Median Incomes and Rent Growth Here We Go Again Bud Clarke, MAI Director, MMA Investment

Median Incomes and Rent Growth Here We Go Again Bud Clarke, MAI Director, MMA Investment Valuation April 7, 2009 The Basics HUD Median Family Incomes (MFI) estimated once a year and published in early March. Section 42 LIHTC

493 views • 20 slides
Selection of Mode of Presentation Selection of the mode of presentation online does not confirm

Selection of Mode of Presentation Selection of the mode of presentation online does not confirm

Selection of Mode of Presentation Selection of the mode of presentation online does not confirm your choice. The Dental Students Scientific Committee reserves the right to determine the mode of selection for each group. However, your preferred

263 views • 3 slides
Median Finding 1. Testing iroot 2. Analyze backboneSimilar 3. Median finding Testing iroot

Median Finding 1. Testing iroot 2. Analyze backboneSimilar 3. Median finding Testing iroot

Median Finding 1. Testing iroot 2. Analyze backboneSimilar 3. Median finding Testing iroot on interval 1, 2, 3, 4, 5 suppose function values for some procedure f are 7, -2, -8, 5, -3 checkExpect(iroot(1, 4, f), ???) ? let k =

553 views • 28 slides

More recommend