introd u ction to e x plorator y data anal y sis
play

Introd u ction to E x plorator y Data Anal y sis STATISTIC AL TH IN - PowerPoint PPT Presentation

Jul 06, 2023 •269 likes •734 views

Introd u ction to E x plorator y Data Anal y sis STATISTIC AL TH IN K IN G IN P YTH ON ( PAR T 1 ) J u stin Bois Lect u rer at the California Instit u te of Technolog y E x plorator y data anal y sis The process of organi z ing , plo ing ,

  1. Introd u ction to E x plorator y Data Anal y sis STATISTIC AL TH IN K IN G IN P YTH ON ( PAR T 1 ) J u stin Bois Lect u rer at the California Instit u te of Technolog y

  2. E x plorator y data anal y sis The process of organi z ing , plo � ing , and s u mmari z ing a data set STATISTICAL THINKING IN PYTHON ( PART 1)

  3. “ E x plorator y data anal y sis can ne v er be the w hole stor y, b u t nothing else can ser v e as the fo u ndation stone .” — John T u ke y STATISTICAL THINKING IN PYTHON ( PART 1)

  4. 2008 US s w ing state election res u lts 1 Data retrie v ed from Data . go v ( h � ps ://www. data . go v/) STATISTICAL THINKING IN PYTHON ( PART 1)

  5. 2008 US s w ing state election res u lts import pandas as pd df_swing = pd.read_csv('2008_swing_states.csv') df_swing[['state', 'county', 'dem_share']] state county dem_share 0 PA Erie County 60.08 1 PA Bradford County 40.64 2 PA Tioga County 36.07 3 PA McKean County 41.21 4 PA Potter County 31.04 5 PA Wayne County 43.78 6 PA Susquehanna County 44.08 7 PA Warren County 46.85 8 OH Ashtabula County 56.94 1 Data retrie v ed from Data . go v ( h � ps ://www. data . go v/) STATISTICAL THINKING IN PYTHON ( PART 1)

  6. 2008 US s w ing state election res u lts 1 Data retrie v ed from Data . go v ( h � ps ://www. data . go v/) STATISTICAL THINKING IN PYTHON ( PART 1)

  7. Let ' s practice ! STATISTIC AL TH IN K IN G IN P YTH ON ( PAR T 1 )

  8. Plotting a histogram STATISTIC AL TH IN K IN G IN P YTH ON ( PAR T 1 ) J u stin Bois Lect u rer at the California Instit u te of Technolog y

  9. 2008 US s w ing state election res u lts Data retrie v ed from Data . go v ( h � ps ://www. data . go v/ ) STATISTICAL THINKING IN PYTHON ( PART 1)

  10. Generating a histogram import matplotlib.pyplot as plt _ = plt.hist(df_swing['dem_share']) _ = plt.xlabel('percent of vote for Obama') _ = plt.ylabel('number of counties') plt.show() STATISTICAL THINKING IN PYTHON ( PART 1)

  11. Al w a y s label y o u r a x es STATISTICAL THINKING IN PYTHON ( PART 1)

  12. 2008 US s w ing state election res u lts Data retrie v ed from Data . go v ( h � ps ://www. data . go v/ ) STATISTICAL THINKING IN PYTHON ( PART 1)

  13. Histograms w ith different binning Data retrie v ed from Data . go v ( h � ps ://www. data . go v/ ) STATISTICAL THINKING IN PYTHON ( PART 1)

  14. Setting the bins of a histogram bin_edges = [0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100] _ = plt.hist(df_swing['dem_share'], bins=bin_edges) plt.show() STATISTICAL THINKING IN PYTHON ( PART 1)

  15. Setting the bins of a histogram _ = plt.hist(df_swing['dem_share'], bins=20) plt.show() STATISTICAL THINKING IN PYTHON ( PART 1)

  16. Seaborn An e x cellent Matplotlib - based statistical data v is u ali z ation package w ri � en b y Michael Waskom STATISTICAL THINKING IN PYTHON ( PART 1)

  17. Setting Seaborn st y ling import seaborn as sns sns.set() _ = plt.hist(df_swing['dem_share']) _ = plt.xlabel('percent of vote for Obama') _ = plt.ylabel('number of counties') plt.show() STATISTICAL THINKING IN PYTHON ( PART 1)

  18. A Seaborn - st y led histogram 1 Data retrie v ed from Data . go v ( h � ps ://www. data . go v/) STATISTICAL THINKING IN PYTHON ( PART 1)

  19. Let ' s practice ! STATISTIC AL TH IN K IN G IN P YTH ON ( PAR T 1 )

  20. Plot all of y o u r data : Bee s w arm plots STATISTIC AL TH IN K IN G IN P YTH ON ( PAR T 1 ) J u stin Bois Lect u rer at the California Instit u te of Technolog y

  21. 2008 US s w ing state election res u lts 1 Data retrie v ed from Data . go v ( h � ps ://www. data . go v/) STATISTICAL THINKING IN PYTHON ( PART 1)

  22. 2008 US s w ing state election res u lts 1 Data retrie v ed from Data . go v ( h � ps ://www. data . go v/) STATISTICAL THINKING IN PYTHON ( PART 1)

  23. Binning bias The same data ma y be interpreted di � erentl y depending on choice of bins STATISTICAL THINKING IN PYTHON ( PART 1)

  24. Bee s w arm plot 1 Data retrie v ed from Data . go v ( h � ps ://www. data . go v/) STATISTICAL THINKING IN PYTHON ( PART 1)

  25. Organi z ation of the data frame 1 Data retrie v ed from Data . go v ( h � ps ://www. data . go v/) STATISTICAL THINKING IN PYTHON ( PART 1)

  26. Organi z ation of the data frame STATISTICAL THINKING IN PYTHON ( PART 1)

  27. Organi z ation of the data frame STATISTICAL THINKING IN PYTHON ( PART 1)

  28. Generating a bee s w arm plot _ = sns.swarmplot(x='state', y='dem_share', data=df_swing) _ = plt.xlabel('state') _ = plt.ylabel('percent of vote for Obama') plt.show() STATISTICAL THINKING IN PYTHON ( PART 1)

  29. 2008 US s w ing state election res u lts 1 Data retrie v ed from Data . go v ( h � ps ://www. data . go v/) STATISTICAL THINKING IN PYTHON ( PART 1)

  30. Let ' s practice ! STATISTIC AL TH IN K IN G IN P YTH ON ( PAR T 1 )

  31. Plot all of y o u r data : ECDFs STATISTIC AL TH IN K IN G IN P YTH ON ( PAR T 1 ) J u stin Bois Lect u rer at the California Instit u te of Technolog y

  32. 2008 US s w ing state election res u lts 1 Data retrie v ed from Data . go v ( h � ps ://www. data . go v/) STATISTICAL THINKING IN PYTHON ( PART 1)

  33. 2008 US election res u lts : East and West 1 Data retrie v ed from Data . go v ( h � ps ://www. data . go v/) STATISTICAL THINKING IN PYTHON ( PART 1)

  34. Empirical c u m u lati v e distrib u tion f u nction ( ECDF ) 1 Data retrie v ed from Data . go v ( h � ps ://www. data . go v/) STATISTICAL THINKING IN PYTHON ( PART 1)

  35. Empirical c u m u lati v e distrib u tion f u nction ( ECDF ) 1 Data retrie v ed from Data . go v ( h � ps ://www. data . go v/) STATISTICAL THINKING IN PYTHON ( PART 1)

  36. Empirical c u m u lati v e distrib u tion f u nction ( ECDF ) 1 Data retrie v ed from Data . go v ( h � ps ://www. data . go v/) STATISTICAL THINKING IN PYTHON ( PART 1)

  37. Making an ECDF import numpy as np x = np.sort(df_swing['dem_share']) y = np.arange(1, len(x)+1) / len(x) _ = plt.plot(x, y, marker='.', linestyle='none') _ = plt.xlabel('percent of vote for Obama') _ = plt.ylabel('ECDF') plt.margins(0.02) # Keeps data off plot edges plt.show() STATISTICAL THINKING IN PYTHON ( PART 1)

  38. 2008 US s w ing state election ECDF 1 Data retrie v ed from Data . go v ( h � ps ://www. data . go v/) STATISTICAL THINKING IN PYTHON ( PART 1)

  39. 2008 US s w ing state election ECDFs 1 Data retrie v ed from Data . go v ( h � ps ://www. data . go v/) STATISTICAL THINKING IN PYTHON ( PART 1)

  40. Let ' s practice ! STATISTIC AL TH IN K IN G IN P YTH ON ( PAR T 1 )

  41. On w ard to w ard the w hole stor y! STATISTIC AL TH IN K IN G IN P YTH ON ( PAR T 1 ) J u stin Bois Lect u rer at the California Instit u te of Technolog y

  42. STATISTICAL THINKING IN PYTHON ( PART 1)

  43. “ E x plorator y data anal y sis can ne v er be the w hole stor y, b u t nothing else can ser v e as the fo u ndation stone .” — John T u ke y STATISTICAL THINKING IN PYTHON ( PART 1)

  44. Coming u p … Thinking probabilisticall y Discrete and contin u o u s distrib u tions The po w er of hacker statistics u sing np.random STATISTICAL THINKING IN PYTHON ( PART 1)

  45. Let ' s practice ! STATISTIC AL TH IN K IN G IN P YTH ON ( PAR T 1 )

Recommend

E x plorator y data anal y sis P R E D IC TIN G C TR W ITH MAC H IN E L E AR N IN G IN P YTH

E x plorator y data anal y sis P R E D IC TIN G C TR W ITH MAC H IN E L E AR N IN G IN P YTH

E x plorator y data anal y sis P R E D IC TIN G C TR W ITH MAC H IN E L E AR N IN G IN P YTH ON Ke v in H u o Instr u ctor A closer look at feat u res int : an integer : 1 , 2 , etc . print(df.columns) float : decimals : 3.02 , 4.56 , etc .

189 views • 18 slides
Introd u ction : Working With Web Data in R W OR K IN G W ITH W E B DATA IN R Oli v er Ke y

Introd u ction : Working With Web Data in R W OR K IN G W ITH W E B DATA IN R Oli v er Ke y

Introd u ction : Working With Web Data in R W OR K IN G W ITH W E B DATA IN R Oli v er Ke y es & Charlo e Wickham Instr u ctors Working w ith Web Data in R Do w nloading les and u sing speciali z ed packages to get data from w eb

554 views • 16 slides
Introd u ction to signals FIN AN C IAL TR AD IN G IN R Il y a Kipnis Professional Q u antitati

Introd u ction to signals FIN AN C IAL TR AD IN G IN R Il y a Kipnis Professional Q u antitati

Introd u ction to signals FIN AN C IAL TR AD IN G IN R Il y a Kipnis Professional Q u antitati v e Anal y st and R programmer What are signals ? Signals are the interactions of : Market data w ith indicators Indicators w ith other indicators E

523 views • 24 slides
Introd u ction to Te x t Encoding FE ATU R E E N G IN E E R IN G FOR MAC H IN E L E AR N IN G

Introd u ction to Te x t Encoding FE ATU R E E N G IN E E R IN G FOR MAC H IN E L E AR N IN G

Introd u ction to Te x t Encoding FE ATU R E E N G IN E E R IN G FOR MAC H IN E L E AR N IN G IN P YTH ON Robert O ' Callaghan Director of Data Science , Ordergroo v e Standardi z ing y o u r te x t E x ample of free te x t : Fello w- Citi z

808 views • 46 slides
Introd u ction VISU AL IZIN G G E OSPATIAL DATA IN P YTH ON Mar y v an Valkenb u rg Data

Introd u ction VISU AL IZIN G G E OSPATIAL DATA IN P YTH ON Mar y v an Valkenb u rg Data

Introd u ction VISU AL IZIN G G E OSPATIAL DATA IN P YTH ON Mar y v an Valkenb u rg Data Science Program Manager , Nash v ille So w are School Location 1854 cholera o u tbreak in London 600+ deaths VISUALIZING GEOSPATIAL DATA IN PYTHON Sno

525 views • 26 slides
Introd u ction to iterators P YTH ON DATA SC IE N C E TOOL BOX ( PAR T 2 ) H u go Bo w ne -

Introd u ction to iterators P YTH ON DATA SC IE N C E TOOL BOX ( PAR T 2 ) H u go Bo w ne -

Introd u ction to iterators P YTH ON DATA SC IE N C E TOOL BOX ( PAR T 2 ) H u go Bo w ne - Anderson Data Scientist at DataCamp Iterating w ith a for loop We can iterate o v er a list u sing a for loop employees = ['Nick', 'Lore', 'Hugo'] for

379 views • 25 slides
Introd u ction to a u dio data in P y thon SP OK E N L AN G U AG E P R OC E SSIN G IN P YTH ON

Introd u ction to a u dio data in P y thon SP OK E N L AN G U AG E P R OC E SSIN G IN P YTH ON

Introd u ction to a u dio data in P y thon SP OK E N L AN G U AG E P R OC E SSIN G IN P YTH ON Daniel Bo u rke Machine Learning Engineer / Yo u T u be Creator Dealing w ith a u dio files in P y thon Di erent kinds all of a u dio les mp

463 views • 17 slides
Introd u ction to APIs and JSONs IN TE R ME D IATE IMP OR TIN G DATA IN P YTH ON H u go Bo w

Introd u ction to APIs and JSONs IN TE R ME D IATE IMP OR TIN G DATA IN P YTH ON H u go Bo w

Introd u ction to APIs and JSONs IN TE R ME D IATE IMP OR TIN G DATA IN P YTH ON H u go Bo w ne - Anderson Data Scientist at DataCamp APIs Application Programming Interface Protocols and ro u tines B u ilding and interacting w ith so w

426 views • 24 slides
Introd u ction to the dataset W OR K IN G W ITH G E OSPATIAL DATA IN P YTH ON Joris Van den

Introd u ction to the dataset W OR K IN G W ITH G E OSPATIAL DATA IN P YTH ON Joris Van den

Introd u ction to the dataset W OR K IN G W ITH G E OSPATIAL DATA IN P YTH ON Joris Van den Bossche Open so u rce so w are de v eloper and teacher , GeoPandas maintainer Artisanal mining site data from IPIS IPIS : International Peace

531 views • 38 slides
Introd u ction to s w imming data C ASE STU D IE S IN STATISTIC AL TH IN K IN G J u stin Bois

Introd u ction to s w imming data C ASE STU D IE S IN STATISTIC AL TH IN K IN G J u stin Bois

Introd u ction to s w imming data C ASE STU D IE S IN STATISTIC AL TH IN K IN G J u stin Bois Lect u rer , Caltech The 2015 FINA World Championships 1 Photo b y Chan - Fan , CC - BY - SA -4.0 CASE STUDIES IN STATISTICAL THINKING Strokes at

439 views • 27 slides
Introd u ction to statistical seismolog y C ASE STU D IE S IN STATISTIC AL TH IN K IN G J u

Introd u ction to statistical seismolog y C ASE STU D IE S IN STATISTIC AL TH IN K IN G J u

Introd u ction to statistical seismolog y C ASE STU D IE S IN STATISTIC AL TH IN K IN G J u stin Bois Lect u rer , Caltech California mo v es and shakes 1 Fa u lt data : USGS Q u aternar y Fa u lt and Fold Database of the United States CASE

970 views • 39 slides
Introd u ction to relational databases IN TR OD U C TION TO IMP OR TIN G DATA IN P YTH ON H u

Introd u ction to relational databases IN TR OD U C TION TO IMP OR TIN G DATA IN P YTH ON H u

Introd u ction to relational databases IN TR OD U C TION TO IMP OR TIN G DATA IN P YTH ON H u go Bo w ne - Anderson Data Scientist at DataCamp What is a relational database ? Based on relational model of data First described b y Edgar Ted

634 views • 50 slides
Introd u ction to databases STR E AML IN E D DATA IN G E STION W ITH PAN DAS Aman y Mahfo uz

Introd u ction to databases STR E AML IN E D DATA IN G E STION W ITH PAN DAS Aman y Mahfo uz

Introd u ction to databases STR E AML IN E D DATA IN G E STION W ITH PAN DAS Aman y Mahfo uz Instr u ctor Relational Databases Data abo u t entities is organi z ed into tables Each ro w or record is an instance of an entit y Each col u mn has

718 views • 43 slides
Introd u ction to spreadsheets STR E AML IN E D DATA IN G E STION W ITH PAN DAS Aman y Mahfo

Introd u ction to spreadsheets STR E AML IN E D DATA IN G E STION W ITH PAN DAS Aman y Mahfo

Introd u ction to spreadsheets STR E AML IN E D DATA IN G E STION W ITH PAN DAS Aman y Mahfo uz Instr u ctor Spreadsheets Also kno w n as E x cel les Data stored in tab u lar form , w ith cells arranged in ro w s and col u mns Unlike

976 views • 51 slides
Introd u ction IN TE R ME D IATE IN TE R AC TIVE DATA VISU AL IZATION W ITH P L OTLY IN R

Introd u ction IN TE R ME D IATE IN TE R AC TIVE DATA VISU AL IZATION W ITH P L OTLY IN R

Introd u ction IN TE R ME D IATE IN TE R AC TIVE DATA VISU AL IZATION W ITH P L OTLY IN R Adam Lo y Statistician , Carleton College Moti v ation Is it easier to see the changes o v er time based on the animation ? Or the faceted v ie w s ?

474 views • 31 slides
Introd u ction to Tid y Data W OR K IN G W ITH DATA IN TH E TIDYVE R SE Alison Hill

Introd u ction to Tid y Data W OR K IN G W ITH DATA IN TH E TIDYVE R SE Alison Hill

Introd u ction to Tid y Data W OR K IN G W ITH DATA IN TH E TIDYVE R SE Alison Hill Professor & Data Scientist WORKING WITH DATA IN THE TIDYVERSE WORKING WITH DATA IN THE TIDYVERSE The Great British Bake Off Series 8 WORKING WITH DATA

786 views • 60 slides
Introd u ction to the space la u nches data IN TE R ME D IATE IN TE R AC TIVE DATA VISU AL

Introd u ction to the space la u nches data IN TE R ME D IATE IN TE R AC TIVE DATA VISU AL

Introd u ction to the space la u nches data IN TE R ME D IATE IN TE R AC TIVE DATA VISU AL IZATION W ITH P L OTLY IN R Adam Lo y Statistician , Carleton College La u nches data set Image credit : Airforce Space Command , 45 th Space Wing P u

528 views • 35 slides
Introd u ction to the NASA fireball data set BU IL D IN G DASH BOAR D S W ITH SH IN YDASH BOAR

Introd u ction to the NASA fireball data set BU IL D IN G DASH BOAR D S W ITH SH IN YDASH BOAR

Introd u ction to the NASA fireball data set BU IL D IN G DASH BOAR D S W ITH SH IN YDASH BOAR D L u c y D ' Agostino McGo w an Postdoctoral fello w in Biostatistics at Johns Hopkins Uni v ersit y Data backgro u nd 1 h ps :// cneos . jpl .

174 views • 14 slides
Introd u ction to JSON STR E AML IN E D DATA IN G E STION W ITH PAN DAS Aman y Mahfo uz Instr

Introd u ction to JSON STR E AML IN E D DATA IN G E STION W ITH PAN DAS Aman y Mahfo uz Instr

Introd u ction to JSON STR E AML IN E D DATA IN G E STION W ITH PAN DAS Aman y Mahfo uz Instr u ctor Ja v ascript Object Notation ( JSON ) Common w eb data format Not tab u lar Records don ' t ha v e to all ha v e the same set of a rib u

950 views • 60 slides
Introd u ction to Net w orks IN TR OD U C TION TO N E TW OR K AN ALYSIS IN P YTH ON Eric Ma

Introd u ction to Net w orks IN TR OD U C TION TO N E TW OR K AN ALYSIS IN P YTH ON Eric Ma

Introd u ction to Net w orks IN TR OD U C TION TO N E TW OR K AN ALYSIS IN P YTH ON Eric Ma Data Carpentr y instr u ctor and a u thor of n xv i z package Net w orks ! E x amples : Social Transportation Model relationships bet w een entities

759 views • 39 slides
Introd u ction to animation IN TE R ME D IATE IN TE R AC TIVE DATA VISU AL IZATION W ITH P L

Introd u ction to animation IN TE R ME D IATE IN TE R AC TIVE DATA VISU AL IZATION W ITH P L

Introd u ction to animation IN TE R ME D IATE IN TE R AC TIVE DATA VISU AL IZATION W ITH P L OTLY IN R Adam Lo y Statistician , Carleton College CO 2 emissions and income Q u estion : Has the relationship bet w een carbon dio x ide emissions

453 views • 44 slides
Welcome and Introd u ction SU P E R VISE D L E AR N IN G IN R : R E G R E SSION Nina Z u mel and

Welcome and Introd u ction SU P E R VISE D L E AR N IN G IN R : R E G R E SSION Nina Z u mel and

Welcome and Introd u ction SU P E R VISE D L E AR N IN G IN R : R E G R E SSION Nina Z u mel and John Mo u nt Data Scientists , Win Vector LLC What is Regression ? Regression : Predict a n u merical o u tcome (" dependent v ariable ")

588 views • 30 slides
Introd u ction to Teacher Forcing MAC H IN E TR AN SL ATION IN P YTH ON Th u shan Ganegedara

Introd u ction to Teacher Forcing MAC H IN E TR AN SL ATION IN P YTH ON Th u shan Ganegedara

Introd u ction to Teacher Forcing MAC H IN E TR AN SL ATION IN P YTH ON Th u shan Ganegedara Data Scientist and A u thor The pre v io u s machine translator model The pre v io u s model Encoder GRU Cons u mes English w ords O u tp u ts a conte

637 views • 53 slides
Introd u ction IN TR OD U C TION TO DATA VISU AL IZATION W ITH G G P L OT 2 Rick Sca v e a

Introd u ction IN TR OD U C TION TO DATA VISU AL IZATION W ITH G G P L OT 2 Rick Sca v e a

Introd u ction IN TR OD U C TION TO DATA VISU AL IZATION W ITH G G P L OT 2 Rick Sca v e a Fo u nder , Sca v e a Academ y Yo u r instr u ctor - Rick Sca v etta - e - mail : o ce @ sca v e a . academ y - T w i er : @ Rick _ Sca

577 views • 41 slides

More recommend