data warehousing and machine learning
play

Data Warehousing and Machine Learning Preprocessing Thomas D. - PowerPoint PPT Presentation

Feb 14, 2023 •326 likes •1.18k views

Data Warehousing and Machine Learning Preprocessing Thomas D. Nielsen Aalborg University Department of Computer Science Spring 2008 DWML Spring 2008 1 / 35 Preprocessing Before you can start on the actual data mining, the data may require

  1. Data Warehousing and Machine Learning Preprocessing Thomas D. Nielsen Aalborg University Department of Computer Science Spring 2008 DWML Spring 2008 1 / 35

  2. Preprocessing Before you can start on the actual data mining, the data may require some preprocessing: • Attributes may be redundant. • Values may be missing. • The data contains outliers. • The data is not in a suitable format. • The values appear inconsistent. Garbage in, garbage out DWML Spring 2008 2 / 35

  3. Preprocessing Data Cleaning ID Zip Gander Income Age Marital status Transaction amount 1001 10048 M 75000 C M 5000 1002 J2S7K7 F -40000 40 W 4000 1003 90210 10000000 45 S 7000 1004 6269 M 50000 0 S 1000 1005 55101 F 99999 30 D 3000 DWML Spring 2008 3 / 35

  4. Preprocessing Data Cleaning ID Zip Gander Income Age Marital status Transaction amount 1001 10048 M 75000 C M 5000 1002 J2S7K7 F -40000 40 W 4000 1003 90210 10000000 45 S 7000 1004 6269 M 50000 0 S 1000 1005 55101 F 99999 30 D 3000 – Correct zip code? DWML Spring 2008 3 / 35

  5. Preprocessing Data Cleaning ID Zip Gander Income Age Marital status Transaction amount 1001 10048 M 75000 C M 5000 1002 J2S7K7 F -40000 40 W 4000 1003 90210 10000000 45 S 7000 1004 6269 M 50000 0 S 1000 1005 55101 F 99999 30 D 3000 – Correct zip code? DWML Spring 2008 3 / 35

  6. Preprocessing Data Cleaning ID Zip Gander Income Age Marital status Transaction amount 1001 10048 M 75000 C M 5000 1002 J2S7K7 F -40000 40 W 4000 1003 90210 ?? 10000000 45 S 7000 1004 6269 M 50000 0 S 1000 1005 55101 F 99999 30 D 3000 – Missing value! DWML Spring 2008 3 / 35

  7. Preprocessing Data Cleaning ID Zip Gander Income Age Marital status Transaction amount 1001 10048 M 75000 C M 5000 1002 J2S7K7 F -40000 40 W 4000 1003 90210 10000000 45 S 7000 1004 6269 M 50000 0 S 1000 1005 55101 F 99999 30 D 3000 – Error/outlier! DWML Spring 2008 3 / 35

  8. Preprocessing Data Cleaning ID Zip Gander Income Age Marital status Transaction amount 1001 10048 M 75000 C M 5000 1002 J2S7K7 F -40000 40 W 4000 1003 90210 10000000 45 S 7000 1004 6269 M 50000 0 S 1000 1005 55101 F 99999 30 D 3000 – Error! DWML Spring 2008 3 / 35

  9. Preprocessing Data Cleaning ID Zip Gander Income Age Marital status Transaction amount 1001 10048 M 75000 C M 5000 1002 J2S7K7 F -40000 40 W 4000 1003 90210 10000000 45 S 7000 1004 6269 M 50000 0 S 1000 1005 55101 F 99999 30 D 3000 – Unexpected precision. DWML Spring 2008 3 / 35

  10. Preprocessing Data Cleaning ID Zip Gander Income Age Marital status Transaction amount 1001 10048 M 75000 C M 5000 1002 J2S7K7 F -40000 40 W 4000 1003 90210 10000000 45 S 7000 1004 6269 M 50000 0 S 1000 1005 55101 F 99999 30 D 3000 – Categorical value? DWML Spring 2008 3 / 35

  11. Preprocessing Data Cleaning ID Zip Gander Income Age Marital status Transaction amount 1001 10048 M 75000 C M 5000 1002 J2S7K7 F -40000 40 W 4000 1003 90210 10000000 45 S 7000 1004 6269 M 50000 0 S 1000 1005 55101 F 99999 30 D 3000 – Error/missing value? DWML Spring 2008 3 / 35

  12. Preprocessing Data Cleaning ID Zip Gander Income Age Marital status Transaction amount 1001 10048 M 75000 C M 5000 1002 J2S7K7 F -40000 40 W 4000 1003 90210 10000000 45 S 7000 1004 6269 M 50000 0 S 1000 1005 55101 F 99999 30 D 3000 Other issues: • What are the semantics of the marital status? DWML Spring 2008 3 / 35

  13. Preprocessing Data Cleaning ID Zip Gander Income Age Marital status Transaction amount 1001 10048 M 75000 C M 5000 1002 J2S7K7 F -40000 40 W 4000 1003 90210 10000000 45 S 7000 1004 6269 M 50000 0 S 1000 1005 55101 F 99999 30 D 3000 Other issues: • What are the semantics of the marital status? • What is the unit of measure for the transaction field? DWML Spring 2008 3 / 35

  14. Preprocessing Missing Values In many real world data bases you will be faced with the problem of missing data: Id. Savings Assets Income Credit Risk ($ 1000s) 1 Medium High 75 Good 2 Low Low 50 Bad 3 25 Bad 4 Medium Medium Good 5 Low Medium 100 Good 6 High High 25 Good 7 Low 25 Bad 8 Medium Medium 75 Good By simply discarding the records with missing data we might unintentionally bias the data. DWML Spring 2008 4 / 35

  15. Preprocessing Missing Values Possible strategies for handling missing data: • Use a predefined constant. • Use the mean (for numerical variables) or the mode (for categorical values). • Use a value drawn randomly form the observed distribution. Id. Savings Assets Income Credit Risk ($ 1000s) 1 Medium High 75 Good 2 Low Low 50 Bad 3 25 Bad 4 Medium Medium Good 5 Low Medium 100 Good 6 High High 25 Good 7 Low 25 Bad 8 Medium Medium 75 Good DWML Spring 2008 5 / 35

  16. Preprocessing Missing Values Possible strategies for handling missing data: • Use a predefined constant. • Use the mean (for numerical variables) or the mode (for categorical values). • Use a value drawn randomly form the observed distribution. Id. Savings Assets Income Credit Risk ($ 1000s) 1 Medium High 75 Good 2 Low Low 50 Bad 3 Low 25 Bad 4 Medium Medium Good 5 Low Medium 100 Good 6 High High 25 Good 7 Low 25 Bad 8 Medium Medium 75 Good Both Low and Medium are ’modes’ for savings. DWML Spring 2008 5 / 35

  17. Preprocessing Missing Values Possible strategies for handling missing data: • Use a predefined constant. • Use the mean (for numerical variables) or the mode (for categorical values). • Use a value drawn randomly form the observed distribution. Id. Savings Assets Income Credit Risk ($ 1000s) 1 Medium High 75 Good 2 Low Low 50 Bad 3 Low High 25 Bad 4 Medium Medium Good 5 Low Medium 100 Good 6 High High 25 Good 7 Low Medium 25 Bad 8 Medium Medium 75 Good High and Medium are drawn randomly from the observed distribution for Assets. DWML Spring 2008 5 / 35

  18. Preprocessing Missing Values Possible strategies for handling missing data: • Use a predefined constant. • Use the mean (for numerical variables) or the mode (for categorical values). • Use a value drawn randomly form the observed distribution. Id. Savings Assets Income Credit Risk ($ 1000s) 1 Medium High 75 Good 2 Low Low 50 Bad 3 Low High 25 Bad 4 Medium Medium 54 Good 5 Low Medium 100 Good 6 High High 25 Good 7 Low Medium 25 Bad 8 Medium Medium 75 Good 54 ≈ 75 + 50 + 25 + 100 + 25 + 25 + 75 . 7 DWML Spring 2008 5 / 35

  19. Preprocessing Discretization Some data mining algorithms can only handle discrete attributes. Possible solution: Divide the continuous range into intervals. Example: ( Income , Risk ) = � ( 25 , B ) , ( 25 , B ) , ( 50 , G ) , ( 51 , B ) , ( 54 , G ) , ( 75 , G ) , ( 75 , G )( 100 , G ) , ( 100 , G ) � Unsupervised discretization Equal width binning (width 25): Bin 1: 25, 25 [ 25 , 50 ) Bin 2: 50, 51, 54 [ 50 , 75 ) Bin 3: 75, 75, 100, 100 [ 75 , 100 ] Equal frequency binning (bin density 3): Bin 1: 25, 25, 50 [ 25 , 50 . 5 ) Bin 2: 51, 54, 75, 75 [ 50 . 5 , 87 . 5 ) Bin 3: 100, 100 [ 87 . 5 , 100 ] DWML Spring 2008 6 / 35

  20. Preprocessing Supervised discretization Take the class distribution into account when selecting the intervals. For example, recursively bisect the interval by selecting the split point giving the highest information gain: » | S ≤ v | | S | Ent ( S ≤ v ) + | S > v | – Gain ( S , v ) = Ent ( S ) − | S | Ent ( S > v ) Until some stopping criteria is met. ( Income , Risk ) = � ( 25 , B ) , ( 25 , B ) , ( 50 , G ) , ( 51 , B ) , ( 54 , G ) , ( 75 , G ) , ( 75 , G )( 100 , G ) , ( 100 , G ) � „ 3 3 9 + 6 6 « Ent ( S ) = − 9 log 2 9 log 2 = 0 . 9183 9 Split E-Ent Interval 25 0 . 4602 ( −∞ , 25 ] , ( 25 , ∞ ) 50 0 . 7395 ( −∞ , 50 ] , ( 50 , ∞ ) 51 0 . 3606 ( −∞ , 51 ] , ( 51 , ∞ ) 54 0 . 5394 ( −∞ , 54 ] , ( 54 , ∞ ) 75 0 . 7663 ( −∞ , 75 ] , ( 75 , ∞ ) DWML Spring 2008 7 / 35

  21. Preprocessing Data Transformation Some data mining tools tends to give variables with a large range a higher significance than variables with a smaller range. For example, • Age versus income. DWML Spring 2008 8 / 35

  22. Preprocessing Data Transformation Some data mining tools tends to give variables with a large range a higher significance than variables with a smaller range. For example, • Age versus income. The typical approach is to standardize the scales: 1 0.8 Min-Max Normalization: normalized values 0.6 X − min ( X ) X ∗ = max ( X ) − min ( X ) . 0.4 0.2 A1 A2 0 -20 0 20 40 60 80 100 120 original values DWML Spring 2008 8 / 35

Recommend

Data Warehousing and Machine Learning Probabilistic Classifiers Thomas D. Nielsen Aalborg

Data Warehousing and Machine Learning Probabilistic Classifiers Thomas D. Nielsen Aalborg

Data Warehousing and Machine Learning Probabilistic Classifiers Thomas D. Nielsen Aalborg University Department of Computer Science Spring 2008 DWML Spring 2008 1 / 34 Probabilistic Classifiers Conditional class probabilities Id. Savings

761 views • 59 slides
Data Warehousing and Machine Learning Feature Selection Thomas D. Nielsen Aalborg University

Data Warehousing and Machine Learning Feature Selection Thomas D. Nielsen Aalborg University

Data Warehousing and Machine Learning Feature Selection Thomas D. Nielsen Aalborg University Department of Computer Science Spring 2008 DWML Spring 2008 1 / 16 Feature Selection When features dont help Data generated by process

578 views • 31 slides
Data Warehousing and Machine Learning Density-based clustering Thomas D. Nielsen Aalborg

Data Warehousing and Machine Learning Density-based clustering Thomas D. Nielsen Aalborg

Data Warehousing and Machine Learning Density-based clustering Thomas D. Nielsen Aalborg University Department of Computer Science Spring 2008 Density-based Clustering DWML Spring 2008 1 / 29 Densitiy Based Clustering DBSCAN Idea:

880 views • 54 slides
Data Warehousing Outline Overview of data warehousing Dimensional Modeling Online

Data Warehousing Outline Overview of data warehousing Dimensional Modeling Online

Data Warehousing Outline Overview of data warehousing Dimensional Modeling Online Analytical Processing From OLTP to the Data Warehouse Traditionally, database systems stored data relevant to current business processes

414 views • 14 slides
Database Management Objectives of Lecture 5 Systems Data Warehousing and OLAP Data Warehousing

Database Management Objectives of Lecture 5 Systems Data Warehousing and OLAP Data Warehousing

Lecture 5 Database Management Objectives of Lecture 5 Systems Data Warehousing and OLAP Data Warehousing and OLAP Realize the purpose of data warehousing. Winter 2004 Comprehend the data structures behind data CMPUT 391: Data

591 views • 12 slides
Differential Privacy Machine Learning Li Xiong Big Data + Machine Learning + Machine

Differential Privacy Machine Learning Li Xiong Big Data + Machine Learning + Machine

CS573 Data Privacy and Security Differential Privacy Machine Learning Li Xiong Big Data + Machine Learning + Machine Learning Under Adversarial Settings Data privacy/confidentiality attacks membership attacks, model inversion

3.44k views • 63 slides
Introduction to machine learning COMS 4721 Learning from data Machine learning : the study

Introduction to machine learning COMS 4721 Learning from data Machine learning : the study

Introduction to machine learning COMS 4721 Learning from data Machine learning : the study of computational mechanisms that learn from data in order to make predictions and decisions. Example 1: image classification A birdwatcher

932 views • 14 slides
An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning http://www.cs.iastate.edu/~cs573x/bbsilab.html Machine Learning Software Preparing Data Building Classifiers Interpreting Results Machine Learning Software

496 views • 26 slides
1 EXTERNAL DATA SOURCES Data Warehousing Integrated data spanning EXTRACT TRANSFORM long

1 EXTERNAL DATA SOURCES Data Warehousing Integrated data spanning EXTRACT TRANSFORM long

Data Warehousing and Decision Support Chapter 23, Part A Database Management Systems, 2 nd Edition. R. Ramakrishnan and J. Gehrke 1 Introduction Increasingly, organizations are analyzing current and historical data to identify useful

395 views • 8 slides
Machine Learning Track Data Analytics, Machine Learning and HPC in todays changing

Machine Learning Track Data Analytics, Machine Learning and HPC in todays changing

Intel HPC Developer Convention Salt Lake City 2016 Machine Learning Track Data Analytics, Machine Learning and HPC in todays changing application environment Franz J. Kirly (practical) An overview of data analytics DATA Scientific

550 views • 27 slides
4/14/20 Outline 0) Course Info CS520 1) Introduction Data Integration, Warehousing, and 2)

4/14/20 Outline 0) Course Info CS520 1) Introduction Data Integration, Warehousing, and 2)

4/14/20 Outline 0) Course Info CS520 1) Introduction Data Integration, Warehousing, and 2) Data Preparation and Cleaning Provenance 3) Schema matching and mapping 4) Virtual Data Integration 6. Data Warehousing 5) Data Exchange 6) Data

648 views • 15 slides
Better Machine Learning Through Data Sa Saleema ema Amershi shi Machine T eaching Group

Better Machine Learning Through Data Sa Saleema ema Amershi shi Machine T eaching Group

Better Machine Learning Through Data Sa Saleema ema Amershi shi Machine T eaching Group Microsoft Research August 14, 2016 Making better sense of data. Better data makes better machine learning. Data + Algorithm = Model Data Algorithm

1.16k views • 81 slides
1 EXTERNAL DATA SOURCES Data Warehousing Integrated data spanning EXTRACT long time

1 EXTERNAL DATA SOURCES Data Warehousing Integrated data spanning EXTRACT long time

Data Warehousing and Decision Support [R&G] Chapter 23, Part A CS 432 1 Introduction Increasingly, organizations are analyzing current and historical data to identify useful patterns and support business strategies. Emphasis is on

344 views • 8 slides
5/5/16 Outline 0) Course Info CS520 1) Introduction Data Integration, Warehousing, and 2)

5/5/16 Outline 0) Course Info CS520 1) Introduction Data Integration, Warehousing, and 2)

5/5/16 Outline 0) Course Info CS520 1) Introduction Data Integration, Warehousing, and 2) Data Preparation and Cleaning Provenance 3) Schema matching and mapping 4) Virtual Data Integration 6. Data Warehousing 5) Data Exchange IIT

479 views • 15 slides
Data Management and Analysis with Business Applications Data Warehousing Andrea Brunello

Data Management and Analysis with Business Applications Data Warehousing Andrea Brunello

DMIF, University of Udine Data Management and Analysis with Business Applications Data Warehousing Andrea Brunello andrea.brunello@uniud.it 24th May 2020 Outline 1 Introduction 2 Data Warehousing Fundamental Concepts 3 Data Warehouse General

660 views • 49 slides
x ? Machine Learning 5/4/20 Tim Althoff, UW CS547: Machine Learning for Big Data,

x ? Machine Learning 5/4/20 Tim Althoff, UW CS547: Machine Learning for Big Data,

? ? x ? Machine Learning 5/4/20 Tim Althoff, UW CS547: Machine Learning for Big Data, http://www.cs.washington.edu/cse547 2 ? ? ? ? Machine Learning ? Node classification 5/4/20 Tim Althoff, UW CS547: Machine Learning for Big Data,

755 views • 53 slides
Machine learning for machine data David Andrzejewski - @davidandrzej

Machine learning for machine data David Andrzejewski - @davidandrzej

Machine learning for machine data David Andrzejewski - @davidandrzej Data Sciences, Sumo Logic Strata Conference Machine Data Track February 13, 2014 1

1.1k views • 72 slides
ONLINE MACHINE LEARNING AND DATA MINING EDO LIBERTY STANDARD MACHINE LEARNING SETTING =

ONLINE MACHINE LEARNING AND DATA MINING EDO LIBERTY STANDARD MACHINE LEARNING SETTING =

ONLINE MACHINE LEARNING AND DATA MINING EDO LIBERTY STANDARD MACHINE LEARNING SETTING = call = ? = crawl Training Test Data Data Train call Apply Model Label 2 STANDARD MACHINE LEARNING SETTING Predicting the future

746 views • 51 slides
Machine Learning and Econometrics Hal Varian Jan 2014 Definitions Machine learning, data

Machine Learning and Econometrics Hal Varian Jan 2014 Definitions Machine learning, data

Machine Learning and Econometrics Hal Varian Jan 2014 Definitions Machine learning, data mining, predictive analytics, etc. all use data to predict some variable as a function of other variables. May or may not care about insight,

609 views • 44 slides
Know ledge-Based Systems IS430 Data Warehousing Mostafa Z. Ali Mostafa Z. Ali

Know ledge-Based Systems IS430 Data Warehousing Mostafa Z. Ali Mostafa Z. Ali

Winter 2009 Lesson 6 Know ledge-Based Systems IS430 Data Warehousing Mostafa Z. Ali Mostafa Z. Ali mzali@just.edu.jo Lecture 2: Slide 1 Learning Objectives Understand the basic definitions and concepts of data warehouses Understand

690 views • 56 slides
MACHINE LEARNING Slide adapted from learning from data book and course, and Berkeley cs188 by Dan

MACHINE LEARNING Slide adapted from learning from data book and course, and Berkeley cs188 by Dan

MACHINE LEARNING Slide adapted from learning from data book and course, and Berkeley cs188 by Dan Klein, and Pieter Abbeel Machine Learning ?? Learning from data Tasks: Prediction Classification Recognition Focus on

691 views • 36 slides
Effectiveness of Deep Learning Vs. Machine Learning in a Health Care Use Case RxToDx A Data

Effectiveness of Deep Learning Vs. Machine Learning in a Health Care Use Case RxToDx A Data

Effectiveness of Deep Learning Vs. Machine Learning in a Health Care Use Case RxToDx A Data Science Machine Learning/Deep Learning Show Case Dima Rekesh/Julie Zhu/Ravi Rajagopalan Optum Technology November, 2017 Analytics Machine

483 views • 32 slides
Overview CS 446 What is machine learning? Machine learning : study of computational

Overview CS 446 What is machine learning? Machine learning : study of computational

Overview CS 446 What is machine learning? Machine learning : study of computational mechanisms that learn from data to make predictions and decisions. 1 / 21 Application 1: image classification Birdwatcher takes photos of birds,

1.08k views • 83 slides
Machine Learning: Der KDD-Prozess Knowledge Discovery in Databases Machine Learning Data-Mining

Machine Learning: Der KDD-Prozess Knowledge Discovery in Databases Machine Learning Data-Mining

Machine Learning: Der KDD-Prozess Knowledge Discovery in Databases Machine Learning Data-Mining Aufgaben Aufgabe Von Daten zum Wissen Wissensextraktion durch We are drowing in automatisches Erkennen von information,

1.01k views • 47 slides

More recommend