better together
play

BETTER TOGETHER USING SPARK AND REDSHIFT TO COMBINE YOUR DATA WITH - PowerPoint PPT Presentation

Aug 05, 2023 •233 likes •642 views

BETTER TOGETHER USING SPARK AND REDSHIFT TO COMBINE YOUR DATA WITH PUBLIC DATASETS EUGENE MANDEL (@EUGMANDEL) JAWBONE QCON SF 2014 JAWBONE DATA MOVEMENT SLEEP WORKOUTS MEALS MOOD SOUTH NAPA EARTHQUAKE 2014 % OF PEOPLE AWAKE AT 3:25

  1. BETTER TOGETHER USING SPARK AND REDSHIFT TO COMBINE YOUR DATA WITH PUBLIC DATASETS EUGENE MANDEL (@EUGMANDEL) JAWBONE QCON SF 2014

  2. JAWBONE DATA MOVEMENT SLEEP WORKOUTS MEALS MOOD

  3. SOUTH NAPA EARTHQUAKE 2014

  4. % OF PEOPLE AWAKE AT 3:25 DISTANCE FROM EPICENTER (MILES)

  6. DATA FUSION IS THE PROCESS OF INTEGRATION OF MULTIPLE DATA AND KNOWLEDGE REPRESENTING THE SAME REAL-WORLD OBJECT INTO A CONSISTENT, ACCURATE, AND USEFUL REPRESENTATION. (WIKIPEDIA)

  7. DATA FUSION - HOW TO FIND THE ELEPHANT IMAGE SOURCE: HTTP:/ /COMMONS.WIKIMEDIA.ORG/WIKI/FILE%3ABLIND_MEN_AND_ELEPHANT.PNG

  8. DATA FUSION POWERFUL BUT HARD DATA IS NOISY DOMAIN UNDERSTANDING IS KEY

  9. LET’S TALK ABOUT THE WEATHER

  10. MODEL THE PROBLEM

  11. 7000.0 ACTIVITY 7,000 6700.0 6700.0 6600.0 6500.0 6100.0 ? 6000.0 5,600 5000.0 5000.0 4500.0 4400.0 4300.0 4,200 2,800 1,400 AIR TEMP (°F) 0 0 0 0 0 0 0 0 0 0 0 0 1 2 3 4 5 6 7 8 9 0 1 2 1 1 1

  12. FIND THE DATA

  14. UNDERSTAND THE DATA

  15. HOURLY DAILY

  16. DATA GENERATION PROCESS NETWORK OF WEATHER STATIONS FREQUENCY OF MEASUREMENTS - HOURLY TO DAILY � COLLABORATION WITH INTERNATIONAL AGENCIES � AGGREGATION AND QA BY NCDC �

  17. UNDERSTAND THE DOMAIN WEATHER STATION TIME: 2014-07-09 13:04:00 AIR TEMP: 86°F PRECIPITATION: 3CM

  18. QA THE DATA

  19. BUT ISN’T IT DONE?

  20. …MAYBE NOT! AIR TEMP: 105°F BAKERSFIELD, CA DULUTH, MN JULY 17, 15:00 JAN 12, 05:00

  21. DATA VALIDATION DOMAIN KNOWLEDGE � COMPARE MULTIPLE SOURCES - E.G. CLIMATE � MANUAL REVIEW OF FLAGGED DATA POINTS

  22. JOIN

  23. HOW? DOMAIN SPECIFIC WEATHER STATION A WEATHER STATION B LAT: 39.36 LAT: 39.35 LON: -74.45 LON: -74.44 TIME: 2014-07-09 13:04:00 TIME: 2014-07-09 13:00:00 AIR TEMP: 74°F AIR TEMP: 60°F ELEVATION: 30FT ELEVATION: 120FT

  24. COVERAGE DO THE DATASETS INTERSECT ENOUGH? PLACES � TIMES � USERS

  25. ISOLATE THE EFFECT

  26. CONFOUNDING VARIABLES WHAT ELSE AFFECTS ACTIVITY? WEEKDAYS/WEEKENDS � DAYLIGHT � RAIN/SNOW

  27. REDSHIFT VS SPARK

  28. AMAZON REDSHIFT RELATIONAL ANALYTICAL DATABASE BY AMAZON � COMPLEX QUERIES ON LARGE DATASETS IN SECONDS � SQL INTERFACE (POSTGRES) � MANAGED CLUSTER

  29. EXAMPLE: DAYLIGHT REDSHIFT PYTHON

  30. IN-MEMORY DATA PROCESSING FRAMEWORK � MODELS COMPUTATION AS A GRAPH OF RDDS (RESILIENT DISTRIBUTED DATASETS) � FUNCTIONAL PROGRAMMING MODEL (SCALA, PYTHON) � SQL � CAN READ FROM SAME SOURCES AS HADOOP

  31. EXAMPLE: DAYLIGHT SPARK

  32. SILVER BULLET? PICK YOUR OWN ADVENTURE SPARK REDSHIFT PROGRAMMER-FRIENDLY EASY TO SHARE DATA WITH NON-DEVELOPERS � END-TO-END SOLUTION � MANAGED - EASY SCALING � SELF-DOCUMENTING �

  33. WHAT DID WE FIND?

  34. IDEAL TEMP FOR MOVEMENT DAILY STEPS MAX TEMP (F)

  35. AND NOW BY STATE… DAILY STEPS MAX TEMP (F)

  36. HOURLY STEPS BY AIR TEMP WEEKENDS

  37. LESS CHOICE = SMALLER EFFECT WEEKDAYS

  38. DATA FUSION POWERFUL BUT HARD DATA IS NOISY DOMAIN UNDERSTANDING IS KEY

  39. THANK YOU! @EUGMANDEL WWW.LINKEDIN.COM/IN/EUGENEMANDEL

Recommend

More recommend