evaluation of a machine learning framework to forecast
play

Evaluation of a machine learning framework to forecast storm surge - PowerPoint PPT Presentation

Jul 11, 2023 •123 likes •511 views

Evaluation of a machine learning framework to forecast storm surge Daryl Metters Coastal Impacts Unit Department of Environment and Science The Coastal Impacts: what we do The primary Queensland Government agency involved in the

  1. Evaluation of a machine learning framework to forecast storm surge Daryl Metters Coastal Impacts Unit Department of Environment and Science

  2. The Coastal Impacts: what we do • The primary Queensland Government agency involved in the management of extreme storm-tide events in Queensland • Operate 36 storm-tide gauges and 14 tide gauges along the Queensland coast. These measure the magnitude of storm- tide during cyclonic events for the use by disaster management groups for evacuation purposes • During severe events we Liaise with the Bureau of Meteorology to confirm information in storm-tide advice (warnings) and provide technical advice on storm-tide to local, district and state groups

  3. What is Storm Tide ?

  4. What is Storm Tide ?

  5. What is Storm Surge? Storm Surge = Actual - Tide Prediction

  6. Storm-tide data • We present and distribute the actual storm-tide data along with the tide prediction and residual in near real time • This process predicts or forecasts the tide component only and not the surge level • The surge level is calculated in near real time as the levels are reported by each STG • Storm Surge = actual (measured) level – tide prediction.

  7. Storm Surge forecasting • Many methods developed over recent years to help forecast the storm surge level • Most make use of the linear relationship between the driving parameter(s) and the actual water level recorded • Successful methods used in modern times are based on numerical modelling of the physical driving forces responsible for the surge levels • These modelling efforts are expensive to maintain due to the large computing power needed to operate the models

  8. Storm Surge forecast Important for planning: 1. Evacuation during severe events 2. Recreational activities 3. Commercial transport 4. Scientific marine activities

  9. The Project Storm-tide forecasting using Machine Learning A proof of concept • Transfer of knowledge and understanding of machine learning principles to DES staff • To establish and test various machine learning models, and use those machine learning models to forecast storm-tide levels • Formulate an understanding of the effectiveness of machine learning in forecasting storm-tide

  10. Machine Learning • A type of artificial intelligence • Enables the ability to "learn" with data, without being explicitly programmed • Explores the study and construction of algorithms that can learn from and make predictions on data • This overcomes following strictly static program instructions by making data-driven predictions or decisions, through building a model from sample inputs • Employed in a range of computing tasks where designing and programming explicit algorithms with good performance is difficult or unfeasible.

  11. Computing and code • Python library: Scikit-learn • Anaconda 3 • Used CPU and memory • Amazon AWS High Performance Computing facilities 72 cores of virtual CPUs and 144 GB memory

  12. Data preparation • Storm tide, wind and pressure data checked for errors and gaps • Filtered for out of range values: if out of range then removed • Single missing points given the average of the two points before and after, larger gaps were considered missing data and removed.

  13. Model Training, Testing and Forecasting • The prepared dataset was divided into training and testing datasets • 2/3 training : to improve model accuracy • 1/3 testing : to check model accuracy • The ML model output is then used to forecast 72 hours beyond the training and testing datasets

  14. Input Data • Clump Point Storm Tide Gauge • Storm Tide: one minute • Atmospheric pressure: one minute • Wind speed : 10 minute • Wind direction: 10 minute • Tide predictions : 10 minute

  15. Machine Learning Two general types of machine learning models utilised (1) Feature based models: • Decision Tree • Neural Networks • Linear • k-Nearest Neighbour (kNN) • Support Vector Machine (SVM) and • Random Forest (2) Time series models: ARIMA and Prophet.

  18. Model approach's • V1 Modelled weather approach Storm Tide Training/testing Input : storm tide, atmospheric pressure, wind speed and direction, tide predictions Forecast input : BoM Access-G modelled weather forecast Storm Surge Training/testing Input : residual, atmospheric pressure, wind speed and direction Forecast input : BoM Access-G modelled weather forecast • V2 Dataset shift approach • V3 Modelled weather forecast approach using the time series models • V2 and V3 approach’s not taken to forecast phase

  19. Model performance • Metrics used: 1. Run time: Linux command 'time' used to generate real time: wall clock time - time from start to finish of the call. 2. Mean Square error 3. Correlation coefficients

  20. Model performance Model Run Time Forecast type Model Real time Storm Tide Decision Tree 0m 9.5s Neural Network 128m 7.9s Linear Model 0m 8.7s KNN 0m 25.8s Random Forest 14m 35.8s SVR 31m 23.9s Residual Decision Tree 0m 12.0s Neural Network 132m 45.7s Linear Model 0m 8.6s KNN 0m 56.8s Random Forest 14m 50.4s SVR 31m 29.1s

  21. Model performance Model Run Time Forecast type Model Real time Storm Tide Decision Tree 0m 9.5s Neural Network 128m 7.9s Linear Model 0m 8.7s KNN 0m 25.8s Random Forest 14m 35.8s SVR 31m 23.9s Residual Decision Tree 0m 12.0s Neural Network 132m 45.7s Linear Model 0m 8.6s KNN 0m 56.8s Random Forest 14m 50.4s SVR 31m 29.1s

  22. Model performance Model Run Time Forecast type Model Real time Storm Tide Decision Tree 0m 9.5s Neural Network 128m 7.9s Linear Model 0m 8.7s KNN 0m 25.8s Random Forest 14m 35.8s SVR 31m 23.9s Residual Decision Tree 0m 12.0s Neural Network 132m 45.7s Linear Model 0m 8.6s KNN 0m 56.8s Random Forest 14m 50.4s SVR 31m 29.1s

  23. Model performance testing phase Storm Tide with tide predictions Correlation Model Mean Squared coefficient error 0.988 KNN 0.010 0.990 SVR 0.007 0.990 Decision Tree 0.008 0.991 Random Forest 0.007 0.990 Linear Model 0.007 0.990 Neural Network 0.007 High correlations All models performed equally well

  24. Model performance testing phase Storm Tide without tide predictions Correlation Model Mean Squared coefficient error KNN 0.400 0.057 SVR 0.374 0.069 Decision Tree 0.374 0.111 Random Forest 0.374 0.106 Linear Model 0.377 0.019 Neural Network 0.378 -0.014 Very low correlation All models performed poorly

  25. Model performance testing phase Storm Surge (Residual) 1 month 3 months 6 months 12 months average Model 0.264 0.288 0.421 0.398 0.343 KNN 0.377 0.398 0.568 0.403 0.437 SVR 0.307 0.326 0.400 0.425 0.365 Decision Tree 0.375 0.383 0.610 0.543 0.478 Random Forest 0.400 0.397 0.383 0.343 0.381 Linear Model 0.400 0.393 0.514 0.512 0.455 Neural Network 0.354 0.364 0.483 0.437 average Moderate correlation

  26. Model performance testing phase Storm Surge (Residual) 1 month 3 months 6 months 12 months average Model 0.264 0.288 0.421 0.398 0.343 KNN 0.377 0.398 0.568 0.403 0.437 SVR 0.307 0.326 0.400 0.425 0.365 Decision Tree 0.375 0.383 0.610 0.543 0.478 Random Forest 0.400 0.397 0.383 0.343 0.381 Linear Model 0.400 0.393 0.514 0.512 0.455 Neural Network 0.354 0.364 0.483 0.437 average Moderate correlation Increasing correlation with increase in data length

  27. Model performance testing phase Storm Surge (Residual) 1 month 3 months 6 months 12 months average Model 0.264 0.288 0.421 0.398 0.343 KNN 0.377 0.398 0.568 0.403 0.437 SVR 0.307 0.326 0.400 0.425 0.365 Decision Tree 0.375 0.383 0.610 0.543 0.478 Random Forest 0.400 0.397 0.383 0.343 0.381 Linear Model 0.400 0.393 0.514 0.512 0.455 Neural Network 0.354 0.364 0.483 0.437 average Moderate correlation Increasing correlation with increase in data length Random Forest best performing model

  28. Model performance testing phase Storm Surge (Residual) 1 month 3 months 6 months 12 months average Model 0.264 0.288 0.421 0.398 0.343 KNN 0.377 0.398 0.568 0.403 0.437 SVR 0.307 0.326 0.400 0.425 0.365 Decision Tree 0.375 0.383 0.610 0.543 0.478 Random Forest 0.400 0.397 0.383 0.343 0.381 Linear Model 0.400 0.393 0.514 0.512 0.455 Neural Network 0.354 0.364 0.483 0.437 average Moderate correlation Increasing correlation with increase in data length Random Forest and Neural Network best performing

  29. Model performance testing phase Time series models Correlation ARIMA model Mean Squared Coefficient error -0.02 Storm Tide 0.438 0.09 Wind Speed 36.739 0.13 Wind Direction 6438.297 0.51 Air Pressure 67.119 Correlation Prophet model Mean Squared Coefficient error 0.34 Storm Tide 0.014 0.13 Wind Speed 86.645 0.41 Wind Direction 8958.065 0.59 Air Pressure 54.353 Very low correlation for Storm Tide, and wind speed and direction

Recommend

A framework for comparative evaluation of different models of Interprofessional Learning

A framework for comparative evaluation of different models of Interprofessional Learning

Professor Maree OKeefe A framework for comparative evaluation of different models of Interprofessional Learning University of Adelaide 1 Australian Government Office for Learning and Teaching (OLT)/ALTC Has provide key support through:

494 views • 13 slides
Machine Learning in Reservoir Production Simulation and Forecast Serge A. Terekhov NeurOK

Machine Learning in Reservoir Production Simulation and Forecast Serge A. Terekhov NeurOK

Machine Learning in Reservoir Production Simulation and Forecast Serge A. Terekhov NeurOK Techsoft, LLC, Moscow, Russia email : serge.terekhov@gmail.com Scaling Up and Modeling for Transport and Flow in Porous Media Dubrovnik, Croatia, 13-16

473 views • 14 slides
TABLA: A Framework for Accelerating Statistical Machine Learning Presenters: MeiXing Dong,

TABLA: A Framework for Accelerating Statistical Machine Learning Presenters: MeiXing Dong,

TABLA: A Framework for Accelerating Statistical Machine Learning Presenters: MeiXing Dong, Lajanugen Logeswaran Intro Machine learning algorithms widely used, computationally CAT intensive FPGAs get performance gains w/

525 views • 17 slides
Machine Learning Basics Prof. Kuan-Ting Lai 2020/4/4 Machine Learning Francois Chollet , Deep

Machine Learning Basics Prof. Kuan-Ting Lai 2020/4/4 Machine Learning Francois Chollet , Deep

Machine Learning Basics Prof. Kuan-Ting Lai 2020/4/4 Machine Learning Francois Chollet , Deep Learning with Python, Manning, 2017 Machine Learning Flow ( ) ( ) ( ) Data Evaluation Training

783 views • 62 slides
End to End Framework for Developing Machine Learning Solution Presenter: Karan Vijay Singh

End to End Framework for Developing Machine Learning Solution Presenter: Karan Vijay Singh

End to End Framework for Developing Machine Learning Solution Presenter: Karan Vijay Singh Outline Introduction to Machine Learning Types of Learning Why designing a Machine Learning Solution is different from designing a

364 views • 23 slides
Introduction to Machine Learning Evaluation: Simple Measures for Classification Learning goals

Introduction to Machine Learning Evaluation: Simple Measures for Classification Learning goals

Introduction to Machine Learning Evaluation: Simple Measures for Classification Learning goals Know the definitions of misclassification error rate (MCE) and accuracy (ACC) Understand the entries of a confusion matrix Understand the idea of

704 views • 10 slides
Introduction to Machine Learning Evaluation: Measures for Regression Learning goals Know the

Introduction to Machine Learning Evaluation: Measures for Regression Learning goals Know the

Introduction to Machine Learning Evaluation: Measures for Regression Learning goals Know the definitions of mean squared error (MSE) and mean absolute error (MAE) Understand the connections of MSE and MAE to L2 and L1 loss Know the

416 views • 5 slides
A Machine Learning based Framework for Building Application Failure Prediction Models

A Machine Learning based Framework for Building Application Failure Prediction Models

A Machine Learning based Framework for Building Application Failure Prediction Models Tanjila Ahmed Outline Objective Motivation Why F 2 PM ? F 2 PM Framework Steps to Implement F 2 PM Experimental Setup Results

260 views • 22 slides
Introduction to Machine Learning Evaluation: Resampling compstat-lmu.github.io/lecture_i2ml

Introduction to Machine Learning Evaluation: Resampling compstat-lmu.github.io/lecture_i2ml

Introduction to Machine Learning Evaluation: Resampling compstat-lmu.github.io/lecture_i2ml RESAMPLING Aim: Assess the performance of learning algorithm. Make training sets large (to keep the pessimistic bias small), and reduce variance

778 views • 10 slides
Machine Learning Problem Framework Presenters: Bikramjeet Singh(20752928) Priyansh Narang

Machine Learning Problem Framework Presenters: Bikramjeet Singh(20752928) Priyansh Narang

Machine Learning Problem Framework Presenters: Bikramjeet Singh(20752928) Priyansh Narang (20716980) Agenda Background research Brief introduction to Machine Learning ML Problem: Formulation ML Pipeline Questions

924 views • 39 slides
Chapter 8 Evaluation Statistical Machine Translation Evaluation How good is a given machine

Chapter 8 Evaluation Statistical Machine Translation Evaluation How good is a given machine

Chapter 8 Evaluation Statistical Machine Translation Evaluation How good is a given machine translation system? Hard problem, since many different translations acceptable semantic equivalence / similarity Evaluation metrics

456 views • 42 slides
An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning http://www.cs.iastate.edu/~cs573x/bbsilab.html Machine Learning Software Preparing Data Building Classifiers Interpreting Results Machine Learning Software

496 views • 26 slides
Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by

Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by

Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by which computers can be trained through observation, rather than being explicitly programmed. Basically, a program is given input and adjusts its

559 views • 17 slides
Performance evaluation and hyperparameter tuning of statistical and machine-learning models using

Performance evaluation and hyperparameter tuning of statistical and machine-learning models using

Performance evaluation and hyperparameter tuning of statistical and machine-learning models using spatial data Patrick Schratz 1 , Jannes Muenchow 1 , Eugenia Iturritxa 2 , Jakob Richter 3 , Alexander Brenning 1 September 27 2018, 10th

632 views • 37 slides
Machine Learning: Study of algorithms that improve their performance P at some task T

Machine Learning: Study of algorithms that improve their performance P at some task T

Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 12, 2015 Today: Readings: What is machine learning? The Discipline of ML Decision tree learning Mitchell, Chapter 3

872 views • 29 slides
Distributed GraphLab A Framework for Machine Learning and Data Mining in the Cloud Y. Low, J.

Distributed GraphLab A Framework for Machine Learning and Data Mining in the Cloud Y. Low, J.

Distributed GraphLab A Framework for Machine Learning and Data Mining in the Cloud Y. Low, J. Gonzalez, A. Kyrola, D. Bickson, C. Guestrin, J. Hellerstein By Maciej Biskupiak for R212 Motivation Abstractions of parallel computation are

463 views • 15 slides
Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning

Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning

GCT634/AI613: Musical Applications of Machine Learning (Fall 2020) Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning Pipeline in Classification Tasks A set of hand-designed audio features are selected

401 views • 26 slides
Statistical Machine Learning Lecture 07: Clustering and Evaluation Kristian Kersting TU

Statistical Machine Learning Lecture 07: Clustering and Evaluation Kristian Kersting TU

Statistical Machine Learning Lecture 07: Clustering and Evaluation Kristian Kersting TU Darmstadt Summer Term 2020 K. Kersting based on Slides from J. Peters Statistical Machine Learning Summer Term 2020 1 / 38 Todays Objectives Make

545 views • 38 slides
Statistical Natural Language Processing Machine learning: evaluation ar ltekin

Statistical Natural Language Processing Machine learning: evaluation ar ltekin

Statistical Natural Language Processing Machine learning: evaluation ar ltekin University of Tbingen Seminar fr Sprachwissenschaft Summer Semester 2017 ML evaluation Measuring success/failure in regression Summer Semester 2017

646 views • 27 slides
How Effective was the UK Carbon Tax? A Machine Learning Approach to Policy Evaluation Jan Abrell,

How Effective was the UK Carbon Tax? A Machine Learning Approach to Policy Evaluation Jan Abrell,

How Effective was the UK Carbon Tax? A Machine Learning Approach to Policy Evaluation Jan Abrell, Mirjam Kosch, Sebastian Rausch IAEE, 25.8.2019 Two main questions 1. What was the impact of the UK carbon price support on emissions? 2. How can

523 views • 34 slides
Applications of Machine Learning to Performance Evaluation Daniel Sadoc Menasche 1 Edmundo de

Applications of Machine Learning to Performance Evaluation Daniel Sadoc Menasche 1 Edmundo de

Random walks on graphs Hidden Markov Models and prediction Applications of Machine Learning to Performance Evaluation Daniel Sadoc Menasche 1 Edmundo de Souza e Silva 2 Federal University of Rio de Janeiro 1 Computer Science Department,

1.45k views • 122 slides
TensorFlow: a Framework for Scalable Machine Learning ACM Learning Center, 2016 You probably

TensorFlow: a Framework for Scalable Machine Learning ACM Learning Center, 2016 You probably

TensorFlow: a Framework for Scalable Machine Learning ACM Learning Center, 2016 You probably want to know... What is TensorFlow? Why did we create TensorFlow? How does TensorFlow work? Code: Linear Regression Code: Convolution

820 views • 54 slides
ROC Analysis for Evaluation of Machine Learning Algorithms Larry Holder School of Electrical

ROC Analysis for Evaluation of Machine Learning Algorithms Larry Holder School of Electrical

ROC Analysis for Evaluation of Machine Learning Algorithms Larry Holder School of Electrical Engineering and Computer Science Washington State University References Provost et al., The Case Against Accuracy Estimation for Comparing

339 views • 12 slides
A Framework for System Event Classification and Prediction by Means of Machine Learning Teerat

A Framework for System Event Classification and Prediction by Means of Machine Learning Teerat

A Framework for System Event Classification and Prediction by Means of Machine Learning Teerat Pitakrat, Jonas Grunert, Oliver Kabierschke, Fabian Keller and Andr van Hoorn University of Stuttgart Institute of Software Technology (ISTE)

757 views • 52 slides

More recommend