Near real time assessment of operational oceanography products: - PowerPoint PPT Presentation

Near real time assessment of operational oceanography products: advances in the GOV community, overview of Class 4 intercomparison and multimodel assessment approaches Greg Smith Environment Canada F. Hernandez Mercator-Ocean M. Martin, A. Sellar UK Metoffice GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013 With thanks to the many IV-TT contributors! www.godae-oceanview.org GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

The Intercomparison and Validation Task Team: • Pursues activities developed during GODAE • Develop a framework for comparing outputs of the various operational ocean forecasting systems (OOFS) – leads to improvements to the systems and to the quality of products from those systems – provides a framework for scientific discussions on forecasting system performance assessment, on ocean analysis and validation from numerical simulations – offers a demonstration of the work in GODAE OceanView and has the potential to increase the visibility to the external community GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

The Intercomparison and Validation Task Team Coordinates and promotes the development of scientific validation and intercomparison of operational oceanography systems – Mainly for short term forecast assessment, but benefit for/from reanalysis assessment, and medium range forecast assessment (eg, seasonal forecast), – include the definition of metrics to assess the quality of analyses and forecasts (e.g. forecast skills) both for physical and biogeochemical parameters and the setting up of specific global and regional intercomparison experiments – metrics related to specific applications also considered – links with the OSE-TT – links with the JCOMM ET-OOFS team for operational implementation – cooperation with CLIVAR/GSOP for climate issues. GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Outline of this talk • Main outcomes from the IV-TT: – Class 4 metrics, forecast skill monitored in near real time among 6 operational centres – Class 1 metrics, ensemble and multi-model approach – Participation to the GSOP Intercomparison project (CLIVAR/GOV initiative, presented by M. Balmaseda) • Advances in real time monitoring and validation methodology implemented in operational oceanography centres: highlights with some examples • How operational validation is provided to users in term of Product Quality: some illustration and examples • Conclusions GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Class 4 metrics • Create a real-time system for generating model-equivalents to a common set of observations • Major effort made to ensure consistency in methods and comparisons • Initial comparison: – Jan-Jun 2013, 6 institutions participated • Systems evaluated using: – SST from drifting buoys – Argo temperature and salinity profiles – Sea level anomaly • Statistics compiled for analyses as well as as a function of lead time GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

USGODAE – In-situ drifters Courtesy Andy Ryan, UK Metoffice GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

ARGO – Salinity profiles GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Taylor diagram of SST around Australia • Taylor diagram provides another perspective See poster from Divakaran for details GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Class4 intercomparison • Radar plots of RMS error computed for the period of 1 st January to 30 th September 2013, for SLA, SST and in the layer 0-500m for temperature and salinity • Systems involved : PSY3 (Mercator OCEAN), FOAM (UK Met Office), GIOPS (CMC), RTOFS(NOAA/NCEP), OceanMAPS (Bureau Of Meteorology) • For salinity PSY3 shows the best performance • For temperature, except OceanMAPS (lower resolution except in Australian region), four other systems are very close • FOAM SST is the most accurate system with GIOPS approaching in terms of forecast See poster from C. Regnier for details GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Mapping of RMS error • We are working with the July-August-September 2013 quarter with the five systems • The ocean is divided in regular bins of 2 ° x2 ° • In the left we show the system for which the RMS error is the lowest • In the right the corresponding RMS value is displayed for each bin • Maps exhibit a regional distribution of the error • This kind of map is a way to compare system with a given criterion • Need to cross several criteria for robust conclusion See poster from C. Regnier for details GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Class 1 Multi-model and ensemble validations GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Multimodel ensemble SST validation • Demonstrates value of multimodel ensemble • Results sensitive to method used to calculate ensemble mean: – simple – weighted – kmeans See poster from T. Spindler for details GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

CLASS1: drifter trajectory forecast • How can we best use 7-day fcst 1-day 3-day OOFS to determine the position of the AF447 crash? • Use 15m drifter trajectories to estimate error from 6 ~20 km systems 1-day forecast: in 50% of the • Demonstrates added forecasts, the distance between value of multimodel the drifter and the end point trajectory is less than 20km ensemble GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Evaluating oil-spill dispersion forecasting in the Northern Aegean Sea Highlights the importance of ensemble methods! See poster by S. Sofianos et al. for details GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Sea ice verification: Analysis tendencies Van Woert et al. (2004)  Sea ice forecast verification using analyses is unreliable due to large analysis uncertainty • Patchy coverage, foot-print issues, data reliability • Representation of leads  Only evaluate points where the analysis changes by more than 10%  Pro: Only includes points where we 7day GIOPS forecast error for 2011-03-30 have confidence in ice analysis • Focus on ice edge in particular  Con: Excludes areas of incorrect See poster from G. Smith for details model changes • E.g: coastal polynyas, false alarms along the ice edge GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Sea ice verification : Analysis tendencies Van Woert et al. (2004) GIOPS 7 day forecasts for 2011 Analysis persistence GIOPS • Over a full annual cycle, this method provides a reasonably reliable idea of forecast skill • But what about leads, coastal polynyas and false alarms in the marginal ice zone? GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Contingency table analysis • Comparison with IMS Analyses: – Interactive Multisensor Snow and Ice IMS Ice IMS No ice Mapping System (NOAA-NIC) – Daily Northern Hemisphere ice Forecast Hit ice False analyses on 4km grid (ice/water) Ice Alarm • Evaluation Methodology: Forecast Miss Hit water – Calculate contingency table values No ice using 0.4 ice concentration cutoff • Proportion Correction Ice: PCI = Hit ice / (Hit ice + Miss) [0,1] • Proportion Correction Water: PCW = Hit water / (Hit water + False Alarm) [0,1] GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

Ice? Y IMS N IMS Contingency table verification Y Fcst Hit ice False Alarm Miss Hit N Fcst GIOPS 7 day forecasts for 2011 water Δ PCI ( Fcst – Pers) Skill • Proportion Correct Ice – Ability to forecast ice formation and advection – PCI= Hits ice / (Hits + Misses) – Shows skillful forecasts along most ice edges Δ PCW ( Fcst – Pers) • Proportion Correct Water – PCW= Hits water / (Hits water + False Alarms) – Shows important false alarm rate missing from analysis tendency Error verification See poster from G. Smith for details GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013

BAL MFC: Developing ice edge metrics for the Baltic Sea Courtesy of P. Lagemaa Observed ice concentration obs ice edge Ice edge RMS distance obs2mod mod2obs distance distance min distance for model ice edge each edge point Find minimum distance from modeled ice edge N 1   obs   2 N n n RMSdist D for each observed ice edge cell and vice versa mod i N  i 1   D  d ... d d ... d Take the RMS of these distances i mod 2 obs , 1 mod 2 obs , n _ obs obs 2 mod, 1 obs 2 mod, n _ mod Straightforward interpretation for the users Metric must be calculated sub-regionally to avoid long overland distances GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013 GODAE OceanView Symposium – November 2013

Conclusions • Direct benefit from IV-TT activity: – several OOFS have now implemented routine Class-4 and Class-1 delivery, allowing enhanced monitoring for these teams, and definition of new validation approaches – Operational oceanography validation experts organized similar to that in the atmospheric community GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013