lecture 8 eda
play

Lecture 8: EDA CS109A Introduction to Data Science Pavlos - PowerPoint PPT Presentation

Sep 11, 2022 •376 likes •928 views

Lecture 8: EDA CS109A Introduction to Data Science Pavlos Protopapas, Kevin Rader and Chris Tanner Lecture Outline Data Science Process Example Dataset considerations Comprehensive vs Sampled Biases Visualization Exploration (EDA)

  1. Lecture 8: EDA CS109A Introduction to Data Science Pavlos Protopapas, Kevin Rader and Chris Tanner

  2. Lecture Outline Data Science Process Example Dataset considerations Comprehensive vs Sampled Biases Visualization Exploration (EDA) Communication CS109A, P ROTOPAPAS , R ADER , T ANNER 1

  3. Lecture Outline Data Science Process Example Dataset considerations Comprehensive vs Sampled Biases Visualization Exploration (EDA) Communication CS109A, P ROTOPAPAS , R ADER , T ANNER 2

  4. Example Let’s say that we are interested in the English Premier League (football/soccer) and want to build a model to predict a player’s market value. Does age affect one’s market value? Question CS109A, P ROTOPAPAS , R ADER , T ANNER 3

  5. Example What do we do? CS109A, P ROTOPAPAS , R ADER , T ANNER 4

  6. Example Ask an interesting question What do we do? Get the Data Explore the Data Model the Data Communicate /Visualize the Results CS109A, P ROTOPAPAS , R ADER , T ANNER 5

  7. Dataset Considerations • What data is necessary to answer our question? • Is the source credible/authoritative? (.com, .net, .org, .gov, .name) • How difficult is it to analyze the dataset? (photos, videos, text?) • What is the allowed usage of data under its license? • Who collected the data? • When was the data collected? CS109A, P ROTOPAPAS , R ADER , T ANNER 6

  8. Dataset Considerations (continued) • How was the data collected? • How is the data formatted? • Confidentiality concerns • Does your data collection procedures need to be approved by an IRB? • Comprehensive data vs sampled data? • Biases CS109A, P ROTOPAPAS , R ADER , T ANNER 7

  9. Dataset Considerations (continued) • How was the data collected? • How is the data formatted? • Confidentiality concerns • Does your data collection procedures need to be approved by an IRB? • Comprehensive data vs sampled data? • Biases CS109A, P ROTOPAPAS , R ADER , T ANNER 8

  10. Lecture Outline Data Science Process Example Dataset considerations Comprehensive vs Sampled Biases Visualization Exploration (EDA) Communication CS109A, P ROTOPAPAS , R ADER , T ANNER 9

  11. Dataset Considerations: Comprehensive Data • We have access to all the data points that exist, which is usually 13 million articles a lot • Collected and digitized as part ~500 million tweets per day of generalized procedures of an institution 100,000s votes per year CS109A, P ROTOPAPAS , R ADER , T ANNER 10

  12. Dataset Considerations: Sampled Data • When collecting individual data is relatively expensive • Only a portion of the population is sampled • Not just restricted to polling or surveys CS109A, P ROTOPAPAS , R ADER , T ANNER 11

  13. Lecture Outline Data Science Process Example Dataset considerations Comprehensive vs Sampled Biases Visualization Exploration (EDA) Communication CS109A, P ROTOPAPAS , R ADER , T ANNER 12

  14. Dataset Considerations: Biases • A bias in sampled data occurs when a procedure causes the sample to overrepresent a subpopulation • Biases may not necessarily be intentional • Even if you don’t think over-representation of a subpopulation will bias the dataset with regard to your question, it’s still a bias • Always strive to minimize any biases in your data collection procedures CS109A, P ROTOPAPAS , R ADER , T ANNER 13

  15. Dataset Considerations: Biases Gallup Polls • Randomly calls two groups of ~500 people a day by sampling among all possible phone numbers • For landlines, asks for household member who has the next birthday • Calls people living in all 50 states • Tries to assure 70% cellphone, 30% landlines • Weights data to reflect the demographics of the general population CS109A, P ROTOPAPAS , R ADER , T ANNER 14

  16. Dataset Considerations: Biases IMDb Movie Ratings • Registered users rate films 1-10 stars; they are an overrepresented subpopulation relative to the general population • Registered users who rate movies in their free time further over represents a specific segment of the general population • “ Men Are Sabotaging The Online Reviews Of TV Shows Aimed At Women 1 ” 60% who rated Sex in the City were women. Women gave it a 8.1, men gave it 5.8. • 1 fivethirtyeight.com CS109A, P ROTOPAPAS , R ADER , T ANNER 15

  17. Dataset Considerations: Biases IMDb Movie Ratings CS109A, P ROTOPAPAS , R ADER , T ANNER 16

  18. Dataset Considerations: Biases Yelp Reviews • Registered users rate businesses on a 1-5 star scale • Registered users tend to represent a certain subset of the population (those who are more social media inclined and opinionated) • Customers with extreme experiences are more likely to voice their opinions CS109A, P ROTOPAPAS , R ADER , T ANNER 17

  19. Dataset Considerations: Biases Yelp Reviews CS109A, P ROTOPAPAS , R ADER , T ANNER 18

  20. Dataset Considerations: Biases Yelp Reviews Longwood Medical Harvard Square CS109A, P ROTOPAPAS , R ADER , T ANNER 19

  21. Back to our example… Let’s say that we are interested in the English Premier League (football/soccer) and want to build a model to predict a player’s market value. Does age affect one’s market value? Question CS109A, P ROTOPAPAS , R ADER , T ANNER 20

  22. Example: Get the data age name market value club position Alexis Sanchez Arsenal 28 LW 65 Mesut Ozil AM Arsenal 28 50 GK Arsenal 35 7 Petr Cech Theo Walcott Arsenal RW 28 20 Laurent Koscielny Arsenal 31 CB 22 from www.transfermarkt.us CS109A, P ROTOPAPAS , R ADER , T ANNER 21

  23. Example: Get the data Credible/Trustworthy? • Possibly subjective • market values? from www.transfermarkt.us Sampled data • CS109A, P ROTOPAPAS , R ADER , T ANNER 22

  24. Example age name market value club position Alexis Sanchez Arsenal 28 LW 65 Mesut Ozil AM Arsenal 28 50 GK Arsenal 35 7 Petr Cech Theo Walcott Arsenal RW 28 20 Laurent Koscielny Arsenal 31 CB 22 CS109A, P ROTOPAPAS , R ADER , T ANNER 23

  25. Example: Explore the Data age name market value club position Alexis Sanchez Arsenal 28 LW 65 Mesut Ozil AM Arsenal 28 50 GK Arsenal 35 7 Petr Cech Theo Walcott Arsenal RW 28 20 Laurent Koscielny Arsenal 31 CB 22 Does it contain the necessary information? CS109A, P ROTOPAPAS , R ADER , T ANNER 24

  26. Example: Explore the Data age name market value club position Alexis Sanchez Arsenal 28 LW 65 Mesut Ozil AM Arsenal 28 50 GK Arsenal 35 7 Petr Cech Theo Walcott Arsenal RW 28 20 Laurent Koscielny Arsenal 31 CB 22 Missing data? Imputation needed? CS109A, P ROTOPAPAS , R ADER , T ANNER 25

  27. Example: Explore the Data age name market value club position Alexis Sanchez Arsenal 28 LW 65 Mesut Ozil AM Arsenal 28 50 GK Arsenal 35 7 Petr Cech Theo Walcott Arsenal RW 28 20 Laurent Koscielny Arsenal 31 CB 22 Are the data types okay ( df.dtypes )? Should be casted? CS109A, P ROTOPAPAS , R ADER , T ANNER 26

  28. Example: Explore the Data age name market value club position Alexis Sanchez Arsenal 28 LW 65 Mesut Ozil AM Arsenal 28 50 GK Arsenal 35 7 Petr Cech Theo Walcott Arsenal RW 28 20 Laurent Koscielny Arsenal 31 CB 22 Are the values reasonable? DataFrame.describe() … CS109A, P ROTOPAPAS , R ADER , T ANNER 27

  29. Example: Explore the Data Are the values reasonable? DataFrame.describe() … CS109A, P ROTOPAPAS , R ADER , T ANNER 28

  30. Example: Explore the Data Summary statistics can only reveal so much CS109A, P ROTOPAPAS , R ADER , T ANNER 29

  31. Lecture Outline Data Science Process Example Dataset considerations Comprehensive vs Sampled Biases Visualization Exploration (EDA) Communication CS109A, P ROTOPAPAS , R ADER , T ANNER 30

  32. Visualization Same stats do not imply same graphs Same graphs do not imply same stats CS109A, P ROTOPAPAS , R ADER , T ANNER 31

  33. Visualization CS109A, P ROTOPAPAS , R ADER , T ANNER 32

  34. Visualization CS109A, P ROTOPAPAS , R ADER , T ANNER 33

  35. Visualization What are some questions we could ask? CS109A, P ROTOPAPAS , R ADER , T ANNER 34

  36. Visualization Q: How effective are the antibiotics? CS109A, P ROTOPAPAS , R ADER , T ANNER 35

  37. CS109A, P ROTOPAPAS , R ADER , T ANNER 36

  38. If bacteria is gram If bacteria is gram negative, Neomycin positive, Penicillin & is most effective Neomycin are most effective CS109A, P ROTOPAPAS , R ADER , T ANNER 37

  39. How do the bacteria compare? Wainer & Lysen, “That’s funny...” American Scientist, 2009 Adapted from Brian Schmotzer CS109A, P ROTOPAPAS , R ADER , T ANNER 38

  40. How do the bacteria compare? Wainer & Lysen, “That’s funny...” American Scientist, 2009 Adapted from Brian Schmotzer CS109A, P ROTOPAPAS , R ADER , T ANNER 39

  41. How do the bacteria Not a streptococcus! compare? (realized ~30 years later) Actually a streptococcus! (realized ~20 years later) Wainer & Lysen, “That’s funny...” American Scientist, 2009 Adapted from Brian Schmotzer CS109A, P ROTOPAPAS , R ADER , T ANNER 40

  42. Wainer & Lysen, “That’s funny...” American Scientist, 2009 CS109A, P ROTOPAPAS , R ADER , T ANNER 41

  43. Wainer & Lysen, “That’s funny...” American Scientist, 2009 CS109A, P ROTOPAPAS , R ADER , T ANNER 42

Recommend

Lecture # 5 - Monday, Aug 30th In this lecture I reviewed the previous lecture 4, and then

Lecture # 5 - Monday, Aug 30th In this lecture I reviewed the previous lecture 4, and then

Lecture # 5 - Monday, Aug 30th In this lecture I reviewed the previous lecture 4, and then reviewed the final point of lecture #3: Hennigs logical approach to tree inference is valid as logic, but we have a hard time using it because our data

660 views • 7 slides
Algorithms (2IL15) Lecture 13 Wrap-up lecture 1 TU/e Algorithms (2IL15) Lecture 13

Algorithms (2IL15) Lecture 13 Wrap-up lecture 1 TU/e Algorithms (2IL15) Lecture 13

TU/e Algorithms (2IL15) Lecture 13 Algorithms (2IL15) Lecture 13 Wrap-up lecture 1 TU/e Algorithms (2IL15) Lecture 13 Part I of the course: Optimization Problems we want to find valid solution minimizing cost (or maximizing

904 views • 20 slides
In 2020SP, this lecture and lecture 20 are both optional extra material CS 5412/LECTURE 17 Ken

In 2020SP, this lecture and lecture 20 are both optional extra material CS 5412/LECTURE 17 Ken

In 2020SP, this lecture and lecture 20 are both optional extra material CS 5412/LECTURE 17 Ken Birman LEAVE NO TRACE BEHIND Spring, 2020 HTTP://WWW.CS.CORNELL.EDU/COURSES/CS5412/2020SP 1 THE PRIVACY PUZZLE FOR I O T We have sensors

1.04k views • 65 slides
Recall last lecture ... Lecture 8 Also last lecture: Painter's Algorithm More Hidden Surface

Recall last lecture ... Lecture 8 Also last lecture: Painter's Algorithm More Hidden Surface

Recall last lecture ... Lecture 8 Also last lecture: Painter's Algorithm More Hidden Surface Removal Sort polygons in depth. Use the farthest vertex in each polygon as the sorting key. Efficient Painter - binary space partition (BSP)

563 views • 6 slides
Plan Lecture 1 - String diagrams and symmetric monoidal categories Lecture 2 -

Plan Lecture 1 - String diagrams and symmetric monoidal categories Lecture 2 -

Plan Lecture 1 - String diagrams and symmetric monoidal categories Lecture 2 - Resource-sensitive algebraic theories Lecture 3 - Interacting Hopf monoids and graphical linear algebra Lecture 4 - Signal Flow Graphs and recurrence

1.2k views • 61 slides
Where are we at - Topic overview Lecture 1A: Security requirements/features Lecture 7A

Where are we at - Topic overview Lecture 1A: Security requirements/features Lecture 7A

Where are we at - Topic overview Lecture 1A: Security requirements/features Lecture 7A Threatens Privacy Threatens Try to achieve Lecture 2B: Network threats Lecture 3A: Attacks on Lecture 6A: Security Protocols Web servers, malware

381 views • 25 slides
Lecture Capture Introduction to Lecture Capture Learning Outcomes What will lecture capture

Lecture Capture Introduction to Lecture Capture Learning Outcomes What will lecture capture

ISDS & Digital Business Skills Lecture Capture Introduction to Lecture Capture Learning Outcomes What will lecture capture actually do? What happens before a lecture? What happens during a lecture? What does the button do?

527 views • 18 slides
CSE 158 Lecture 4 Web Mining and Recommender Systems More Classifiers Last lecture How

CSE 158 Lecture 4 Web Mining and Recommender Systems More Classifiers Last lecture How

CSE 158 Lecture 4 Web Mining and Recommender Systems More Classifiers Last lecture How can we predict binary or categorical variables? {0,1}, {True, False} {1, , N} Last lecture Will I purchase this product? (yes) Will I click

1.74k views • 60 slides
CSE 158 Lecture 4 Web Mining and Recommender Systems More Classifiers Last lecture How

CSE 158 Lecture 4 Web Mining and Recommender Systems More Classifiers Last lecture How

CSE 158 Lecture 4 Web Mining and Recommender Systems More Classifiers Last lecture How can we predict binary or categorical variables? {0,1}, {True, False} {1, , N} Last lecture Will I purchase this product? (yes) Will I click

850 views • 46 slides
CSE 158 Lecture 4 Web Mining and Recommender Systems More Classifiers Last lecture How

CSE 158 Lecture 4 Web Mining and Recommender Systems More Classifiers Last lecture How

CSE 158 Lecture 4 Web Mining and Recommender Systems More Classifiers Last lecture How can we predict binary or categorical variables? {0,1}, {True, False} {1, , N} Last lecture Will I purchase this product? (yes) Will I click

839 views • 58 slides
Usability of Programming Languages Lecture 4 - directed by your research interests Lecture

Usability of Programming Languages Lecture 4 - directed by your research interests Lecture

Overview Practical experimental course lectures are only introductory Lecture 1 - theoretical principles classic approaches l h current trends in leading research. Lecture 2 - candidate research methods advantages and

499 views • 15 slides
Introduction to AI & Intelligent Agents This Lecture Chapters 1 and 2 Next Lecture

Introduction to AI & Intelligent Agents This Lecture Chapters 1 and 2 Next Lecture

Introduction to AI & Intelligent Agents This Lecture Chapters 1 and 2 Next Lecture Chapter 3.1 to 3.4 (Please read lecture topic material before and after each lecture on that topic) What is Artificial Intelligence? Thought

636 views • 31 slides
Introduction to Numerical Optimization Biostatistics 615/815 Lecture 14 Lecture 14 Course is

Introduction to Numerical Optimization Biostatistics 615/815 Lecture 14 Lecture 14 Course is

Introduction to Numerical Optimization Biostatistics 615/815 Lecture 14 Lecture 14 Course is More Than Half Done! If you have comments they are very welcome y y Lectures Lecture notes Weekly Homework Midterm

795 views • 42 slides
Lecture 12: Clustering 1 6.0002 LECTURE 12 Re Reading Chapter 23 6.0002 LECTURE 12 2 Mach Ma

Lecture 12: Clustering 1 6.0002 LECTURE 12 Re Reading Chapter 23 6.0002 LECTURE 12 2 Mach Ma

Lecture 12: Clustering 1 6.0002 LECTURE 12 Re Reading Chapter 23 6.0002 LECTURE 12 2 Mach Ma chine e Lea earn rning Paradigm Observe set of examples: training data Infer something about process that generated that data Use

847 views • 36 slides
Lecture Outline Regeltechniek Previous lecture: Stability and transient response. Lecture 4

Lecture Outline Regeltechniek Previous lecture: Stability and transient response. Lecture 4

Lecture Outline Regeltechniek Previous lecture: Stability and transient response. Lecture 4 Basics of Feedback Control Robert Babu ska Today: Steady-state response. Delft Center for Systems and Control Faculty of Mechanical

388 views • 7 slides
Psycholinguistics Lecture 2 By Dr.Chelli Lecture Objectives At the end of this lecture, students

Psycholinguistics Lecture 2 By Dr.Chelli Lecture Objectives At the end of this lecture, students

Psycholinguistics Lecture 2 By Dr.Chelli Lecture Objectives At the end of this lecture, students will be able to: Discuss the different stages in the history of psycholinguistics Identify the scholars in the field of psycholinguistics

589 views • 28 slides
Methodology for Lecture Methodology for Lecture Computer Graphics (Spring 2008) Computer

Methodology for Lecture Methodology for Lecture Computer Graphics (Spring 2008) Computer

Methodology for Lecture Methodology for Lecture Computer Graphics (Spring 2008) Computer Graphics (Spring 2008) Lecture deals with lighting (teapot shaded as in HW1) Some Nate Robbins tutor demos in lecture COMS 4160, Lecture 14: OpenGL

535 views • 5 slides
Lecture Outline Regeltechniek Previous lecture: Nyquist plot and stability criterion. Lecture 11

Lecture Outline Regeltechniek Previous lecture: Nyquist plot and stability criterion. Lecture 11

Lecture Outline Regeltechniek Previous lecture: Nyquist plot and stability criterion. Lecture 11 State-space models and state feedback control (We are finished with frequency domain methods.) Robert Babu ska Today: Delft Center for

452 views • 6 slides
CSE Fall 2014 311 Lecture 1 Lecture 1 Lecture 1: Propositional Logic Lecture 1 Foundations

CSE Fall 2014 311 Lecture 1 Lecture 1 Lecture 1: Propositional Logic Lecture 1 Foundations

CSE 311: Foundations of Computing I CSE Fall 2014 311 Lecture 1 Lecture 1 Lecture 1: Propositional Logic Lecture 1 Foundations of Computing I Fall 2014 Some Perspective About the Course We will study the theory needed for CSE: Computer

242 views • 7 slides
Proteomics Steven Meinhardt Lectures Lecture 1 Introduction review of proteins

Proteomics Steven Meinhardt Lectures Lecture 1 Introduction review of proteins

11/29/2012 Proteomics Steven Meinhardt Lectures Lecture 1 Introduction review of proteins Lecture 2 Gel Electrophoresis Lecture 3 Mass Spectrometry Lecture 4 Programs for MS analysis Lecture 5 MIAPE (Minimum

900 views • 8 slides
Multiphase Modelling in Cancer Helen Byrne Wolfson Centre for Mathematical Biology Mathematical

Multiphase Modelling in Cancer Helen Byrne Wolfson Centre for Mathematical Biology Mathematical

Lecture 1 Lecture 2 Lecture 3 Lecture 4 Multiphase Modelling in Cancer Helen Byrne Wolfson Centre for Mathematical Biology Mathematical Institute University of Oxford CRM, Barcelona, April 2018 Lecture 1 Lecture 2 Lecture 3 Lecture 4

1.79k views • 141 slides
Lecture 1: Neurons Lecture 2: Coding with spikes Lecture 3: Tuning curves and receptive fields

Lecture 1: Neurons Lecture 2: Coding with spikes Lecture 3: Tuning curves and receptive fields

Lecture 1: Neurons Lecture 2: Coding with spikes Lecture 3: Tuning curves and receptive fields Lecture 4: Population vectors To gain a basic understanding of how population vectors are constructed 4 credit students: First assignment is

584 views • 31 slides
Algorithms (2IL15) Lecture 10 NP-Completeness, II 1 TU/e Algorithms (2IL15) Lecture 10

Algorithms (2IL15) Lecture 10 NP-Completeness, II 1 TU/e Algorithms (2IL15) Lecture 10

TU/e Algorithms (2IL15) Lecture 10 Algorithms (2IL15) Lecture 10 NP-Completeness, II 1 TU/e Algorithms (2IL15) Lecture 10 Summary of previous lecture P: class of decision problems that can be solved in polynomial time NP:

496 views • 34 slides
Lecture 1: Bioinformatic Algorithms In this lecture Logistics of the course

Lecture 1: Bioinformatic Algorithms In this lecture Logistics of the course

Lecture 1: Bioinformatic Algorithms In this lecture Logistics of the course Introduction to basic biology which will continue in the following lecture Logistics of the Course Logistics About the Course Christina Boucher: (email)

546 views • 40 slides

More recommend