1 Data science and engineering for local weather forecasts Nikhil - PowerPoint PPT Presentation

1

Data science and engineering for local weather forecasts Nikhil R Podduturi Data {Scientist, Engineer} November, 2016

Agenda About MeteoGroup ● Introduction to weather data ● Problem description ● Data science and weather forecasting ● Engineering ● Verification ● Results ● Questions ● 3

How many of you check weather forecasts frequently? 4

5

Weather data 6

1.5 TB/day 7

Types of data Observations: ● WMO weather stations (e.g: surface, upper-air, ships, drifting buoys, aircrafts etc) ● MeteoGroup measurement network 8

Types of data Observations: ● WMO weather stations (e.g: surface, upper-air, ships, drifting buoys, aircrafts etc) ● MeteoGroup measurement network Satellite data 9

Types of data Observations: ● WMO weather stations (e.g: surface, upper-air, ships, drifting buoys, aircrafts etc) ● MeteoGroup measurement network Satellite data Radar data 10

Types of data Observations: WMO weather stations (e.g: surface, upper-air, ships, drifting buoys, aircrafts ● etc) MeteoGroup measurement network ● Satellite data Radar data User data 11

Types of data Observations: WMO weather stations (e.g: surface, upper-air, ships, drifting buoys, aircrafts ● etc) MeteoGroup measurement network ● Satellite data Radar data User data Numerical weather prediction model data 12

Numerical weather prediction models ● Complex and Multidimensional data 13

Numerical weather prediction models ● Complex and multidimensional data ● 5 NWP models from different providers 14

Numerical weather prediction models ● Complex and multidimensional data ● 5 NWP models from different providers ● Data size per day - 0.5 TB 15

Data science and weather forecasting 16

17

Outcome ● Took 24 hours for 24 hour forecasts ● Grid interval - 736 km ● Poor results 18

MeteoGroup Forecasting system 19

MeteoGroup forecasting system 3 years of Machine learning NWP data Trained Forecasts model model 3 years of Daily NWP observation data data 20

MeteoGroup forecasting system Written in pascal 21

MeteoGroup forecasting system Written in pascal Runs on in house high performance computing cluster 22

MeteoGroup forecasting system Written in pascal Runs on in house high performance computing cluster Limitations ● Hard to maintain ● Not very transparent ● Scalability 23

Problem description 24

Next generation forecasting system ● Cloud based solution 25

Next generation forecasting system ● Cloud based solution ● Transparent 26

Next generation forecasting system ● Cloud based solution ● Transparent ● Scalable 27

Next generation forecasting system ● Cloud based solution ● Transparent ● Scalable ● Improve forecasting accuracy 28

Baseline model Downscale to Interpolate NWP data Linear model location missing values 29

Baseline model Downscale to Interpolate NWP data Linear model location missing values Outcome: ● Very fast ● Poor accuracy ● Multicollinearity 30

Iteration 1 ● Address multicollinearity using feature selection ● Scale the features Downscale to Interpolate Scale Feature NWP data Linear model location missing values features selection 31

Iteration 1 ● Address multicollinearity using feature selection ● Scale the features Downscale to Interpolate Scale Feature NWP data Linear model location missing values features selection Outcome: ● Improved accuracy 32

Iteration 2 ● Model selection between linear and non-linear models ● Advanced feature selection Model selection Advance Downscale to Interpolate Scale (linear and NWP data feature location missing values non-linear features selection models) 33

Iteration 2 ● Model selection between linear and non-linear models ● Advanced feature selection Model selection Advance Downscale to Interpolate Scale (linear and NWP data feature location missing values non-linear features selection models) Outcome: ● On par with existing forecasting system ● Slow training 34

Engineering to scale the product 35

Baseline model engineering (Scikit-learn, NumPy, Keras with TensorFlow) 36

Model engineering (Scikit-learn, NumPy, Keras with TensorFlow) Good: Python ML ecosystem ● Familiarity among the team ● Test driven and Agile Development ● Fail fast ● 37

Model engineering (Scikit-learn, NumPy, Keras with TensorFlow) Good: Python ML ecosystem ● Familiarity among the team ● Test driven and Agile Development ● Fail fast ● Bad: Not scalable ● 38

47000 * 15 * 360 model runs Locations Weather attributes Hours e.g: temperature, wind etc 39

Scaling with Apache Airflow Apache Airflow • By AirBnB • Apache product since early 2016 Directed Acyclic Graph (DAG) Components • UI • Scheduler • Executor(s) 40

Apache Airflow DAG ● Hooks (connections) ● Operators (tasks) ● Schedule ● Dependencies 41

Airflow and Mesos deploy AWS S3 persist Airflow scheduler Mesos cluster 42

Airflow and Mesos Cont Integ deploy Persist AWS S3 Airflow scheduler Mesos cluster 43

Verification 44

Model improvement cycle Deploy DAG Verify model Improve DAG 45

Forecast verification Forecast Engine AWS S3 with models JSON-LD 46

Verification metrics ● Mean absolute error ● Root mean squared error ● Mean error ● Heidke skill score ● Equitable threat score ● Probability density functions ● Error percentiles 47

Mean absolute error for different models (Temperature) 48

Probability distribution function for multiple models (Temperature) 49

Percentile graphs for each model (Temperature)

For demo please stop by MG booth 51

Results Cloud based solution AWS S3, EC2, ElastiCache ● Transparent Scalable Improve forecasting accuracy 52

Results Cloud based solution AWS S3, EC2, ElastiCache ● Transparent Verification microservice ● Scalable Improve forecasting accuracy 53

Results Cloud based solution AWS S3, EC2, ElastiCache ● Transparent Verification microservice ● Scalable Mesos cluster ● Training time a month to 5 hours (approx) ● Improve forecasting accuracy 54

Results Cloud based solution AWS S3, EC2, ElastiCache ● Transparent Verification microservice ● Scalable Mesos cluster ● Training time a month to 5 hours (approx) ● Improve forecasting accuracy On par or better ● 55

Improvements Hyperlocal AWS lambda integration Iterate for more accuracy 56

Questions? 57

We are hiring!

59