Verification of space weather forecasts at the UK Met Office Suzy Bingham, Michael Sharpe, David Jackson, Sophie Murray* *Trinity College Dublin AGU, New Orleans 11 th Dec. 2017 @MetOfficeSpace
Outline • Met Office Space Weather Operations Centre (MOSWOC) • Verification in near real-time o Flare forecasts o Geomagnetic storm forecasts • Summary
Met Office Space Weather Operations Centre (MOSWOC) • 24/7 space weather monitoring since 2014 • Fully integrated in Met Office Operations Centre • 2 space weather forecasters on duty (1 dedicated to space weather) • Provides twice daily forecasts, & timely alerts & warnings • 14 Forecasters, 6 Scientists, 4 Business, IT Developers • National capability supporting: government, military & critical sectors (power, satellite operators, etc) Public webpages: https://www.metoffice.gov.uk/public/weather/space-weather/
MOSWOC twice daily space weather forecast 4 day probability forecasts for: geomagnetic storms, X- ray flares, Solar CME arrival time analysis high energy at Earth predictions protons & Summary high energy for next 4 days electron events
MOSWOC flare forecast Example MOSWOC 4-day probabilistic flare forecast Flare occurred Probability that flare will occur Flare category in past 24h? for each of the next 4 days. Not exceedance, i.e. probability is of M-class occurring, not M-class or above. • Geomagnetic storm forecasts: exceedance, 4 categories
Adapting terrestrial near real-time verification systems MOSWOC probabilistic forecast Used for gale warnings Used for shipping forecast Treat as multi -category Treat each category/level separately Use Area Forecast Verification System (AFVS) to assess highest category predicted in that 24h: Use Warnings Verification System (WVS): Assess forecast resolution: Assess human forecast skill by Relative Operating comparing against observations, & calculating: Characteristic (ROC) plots Ranked Probability Score (RPS) Assess forecast reliability: As a performance benchmark, calculate RPS for a ‘short -term Reliability diagrams climatology’ (flares: based on previous 120 days of obs) Assess forecast performance by comparing RPS MOSWOC to RPS ref : (RPSS: relative improvement of Ranked Probability Skill Score (RPSS) probability forecast over reference forecast in predicting category which observation fell into)
Flare forecast verification results: Ranked Probability Skill Score (RPSS) • Plot updated daily on Met Office system to monitor rolling skill of forecasts compared to reference • X- axis: Feb ‘16 – Oct ’16, Y -axis: RPSS • Crosses: mean RPSS/month for days 1-4 • Vertical lines: 90% bootstrapped with replacement confidence intervals (CIs) • Point averages suggest day 1 forecasts are more accurate than days 2-4 (day 1 is based on model) • Point averages for days 1 & 2 lie above no-skill line, & most of days 3 & 4, suggesting forecasts Rolling monthly performance of MOSWOC flare perform better than reference forecasts compared to reference forecast (frequency of occurrence over preceding 120 days) • But CIs cross no-skill line so no statistically significant evidence that forecasts outperform (RPSS: relative improvement of probability forecast over reference reference in predicting max daily flare class forecast in predicting category which observation fell into) (flares are difficult to forecast!)
Geomagnetic storm forecast verification results: ROC plot Relative Operating Characteristic (ROC) plot: 1.0 • Measures forecast discrimination – ability of forecast to 0.9 discriminate between categories. Plotted with Reliability 0.8 diagram. 0.7 • X-axis: false alarm rate 0.6 • Y-axis: hit rate POD 0.5 • Diagonal: no-skill 0.4 • Gives info on hit rates & false alarm rates expected from 0.3 using different probability thresholds to trigger advisory action 0.2 0.1 • Flexing: e.g. if G3 was predicted but G2 occurred then system gets some reward 0.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 POFD • Points are all above the grey diagonal no-skill line, thus ROC-plot for day-1, G1, geomag storm forecast. indicating that geomag storm forecast has skill at Apr ‘15 – Oct ‘16. discriminating events of G1 or above (better than chance) Green line: standard verification system (no flexing) ( □ ): the flexed including low-misses (+) flexed excluding low-misses POFD=probability of false detection. POD=probability of detection.
Summary of verification results • Terrestrial verification systems have been adapted for flare & geomag storm forecasts: o Updated daily o Provide understanding of forecast performance & skill (RPSS, ROC, Reliability plots) o Benchmark climatology forecast used: based on observations from previous few months, e.g. for flares, the previous 120 day frequency of occurrence • Rolling 12-month analysis using RPSS indicates: o day 1 geomagnetic storm activity forecasts typically perform better than a climatology benchmark o no consistent evidence for flare forecasts (flares are more difficult to predict) • ROC & Reliability plots suggest: forecasts were skilful at identifying M-class flares & geomag storms, although both were over-forecast
Summary of verification results continued • It’s early days for operational space weather verification – not enough X-class flare occurrences since service began to allow verification • Common meteorological verification techniques can be easily adapted for space weather • Near real-time verification is particularly useful in the operational environment
MOSWOC forecast verification papers Work in progress with NASA CCMC & flare forecast centres/modellers: Applying the Met Office verification systems to provide verification of the CCMC Flare Scoreboard
Thank you
Recommend
More recommend
