Decadal Prediction Drift as a Particular Challenge for Verification - PowerPoint PPT Presentation

Mean temperature (NH+SH)/2 [K] 290 ● ●●●● ● ● ●● ● ● ●● ●● ● ● ●●● ● ●●● ● ●●● ●● ● ●●●● ●● ● ●● ●● ● ● 289 ● 288 1960 1970 1980 1990 2000 2010 Decadal Prediction Drift as a Particular Challenge for Verification Henning Rust Institut für Meteorologie, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 , H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 1

Decadal climate prediction . . . . . . provides information about the future evolution of the statistics of regional climate from the output of a numerical model that has been initialized with observations . . . 1 1 cited from Meehl et al. [2014] , H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 2

Decadal climate prediction . . . 2 2 taken from Boer et al. [2016] , H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 3

Initialization with “observations” Hindcast set: initialize every year, 10-yr hindcast each courtesy of Jens Grieger , H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 4

Initialization with “observations” Hindcast set: initialize every year, 10-yr hindcast each Full-field assimilate directly Annual mean global temperature, taken from Smith et al. [2013] , H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 4

Initialization with “observations” Hindcast set: initialize every year, 10-yr hindcast each Full-field Anomaly assimilate directly assimilate anomalies Annual mean global temperature, taken from Smith et al. [2013] , H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 4

Verification of decadal prediction Mean temperature (NH+SH)/2 [K] 290 ● ●●●● ● ●● ● ● ● ●● ● ●● ●●● ● ● ●●● ● ●● ● ●●● ●●●● Drift ●● ● ●● ● ●● 289 ● ● 288 1960 1970 1980 1990 2000 2010 0.2 bias( τ ) x Drift adjustment x x x 0.0 x x x x x x −0.2 1 2 3 4 5 6 7 8 9 10 Re-Calibration , H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 5

A framework , H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 6

A framework Q1: Predictions more accurate due to initialization? Q2: Does ensemble spread appropriately represents uncertainty? , H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 6

Verification: What can we expect? 3 T emporal scales annual/seasonal averages 1yr lead-year 1 4yrs lead-years 2-5, 6-9 8yrs lead-years 2-9 more are preferable 3 Decadal prediction verification framework Goddard et al. [2013] , H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 7

Verification: What can we expect? 3 T emporal scales Spatial scales annual/seasonal averages scale of reference or larger, e.g. 1yr lead-year 1 Temp 5 ◦ × 5 ◦ 4yrs lead-years 2-5, 6-9 Precip 2.5 ◦ × 2.5 ◦ 8yrs lead-years 2-9 more are preferable depends on study 3 Decadal prediction verification framework Goddard et al. [2013] , H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 7

Verifying ensemble predictions: Accuracy of mean Ensemble mean 1 N e � H jτ = H ijτ N e i = 1 H ijτ ens. member i , initialization j , lead-year τ , H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 8

Verifying ensemble predictions: Accuracy of mean Ensemble mean 1 N e � H jτ = H ijτ N e i = 1 H ijτ ens. member i , initialization j , lead-year τ Q1: More accurate due to initialization? MSE H MSESS = 1 − MSE R historicals (no initialization but forcing) as reference , H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 8

Verifying ensemble predictions: Spread Ensemble Spread N e 1 � 2 σ 2 � � H j = H ijτ − H jτ N e − 1 i = 1 , H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 9

Verifying ensemble predictions: Spread Ensemble Spread N e 1 � 2 σ 2 � � H j = H ijτ − H jτ N e − 1 i = 1 Q2: Does ensemble spread appropriately represents uncertainty? CRPS ( N (ˆ H j , σ 2 H ) , O j ) Gaussian hindcast distribution a conditional and unconditional bias adjusted ( ˆ H j ) a Gneiting and Raftery [2007] , H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 9

Verifying ensemble predictions: Spread Ensemble Spread N e 1 � 2 σ 2 � � H j = H ijτ − H jτ N e − 1 i = 1 Q2: Does ensemble spread appropriately represents uncertainty? CRPS H CRPSS = 1 − CRPS R MSE as variance of ref. fore- cast, CRPSS=0 is optimal! , H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 9

Verifying ensemble predictions: Spread Ensemble Spread N e 1 � 2 σ 2 � � H j = H ijτ − H jτ N e − 1 i = 1 Q2: Does ensemble spread appropriately represents uncertainty? CRPS H CRPSS = 1 − CRPS R MSE as variance of ref. fore- cast, CRPSS=0 is optimal! , H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 9

Verifying ensemble predictions: Spread Ensemble Spread N e 1 � 2 σ 2 � � H j = H ijτ − H jτ N e − 1 i = 1 Q2: Does ensemble spread appropriately represents uncertainty? � σ 2 H � CRPS H CRPSS = 1 − LESS = ln MSE CRPS R Additionally, logarithmic MSE as variance of ref. fore- ensemble spread score cast, CRPSS=0 is optimal! e.g. Kadow et al. [2014] , H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 9

Small ensembles and significance MiKlip ensembles baseline0 3 baseline1 10 prototype 15 + 15 preop ≤ 15 Goddard et al. [2013] suggest a bootstrap for significance , H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 10

Small ensembles and significance MiKlip ensembles baseline0 3 baseline1 10 prototype 15 + 15 preop ≤ 15 Goddard et al. [2013] suggest a bootstrap for significance , H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 10

Small ensembles and significance MiKlip ensembles baseline0 3 baseline1 10 prototype 15 + 15 preop ≤ 15 Goddard et al. [2013] suggest a bootstrap for significance , H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 10

Small ensembles and biased scores 4 ❼ bias corrected scores Ferro et al. [2008] ❼ application with RPS Kruschke et al. [2015] ❼ implemented in R -package SpecsVerification (Stefan Siegert) 4 taken from Müller et al. [2005] , H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 11

Mean temperature (NH+SH)/2 [K] 290 ● ●●●● ● ●● ● ● ● ●● ● ●● ● ●●● ● ●●● ● ●●● ●● ● ●●●● ●● ● ●● ●● ● 289 ● ● 288 1960 1970 1980 1990 2000 2010 Drift

“Bias” or mean difference 1 N 1 N � � ME = H j − O j N N j = 1 j = 1 , H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 13

“Bias” or mean difference 1 N 1 N � � ME = H j − O j := b N N j = 1 j = 1 For decadal prediction and other cases b = b ( τ, X ) , τ : forecast lead-time; X and climate state X . Sytematic error we belief we can compensate a posteriori , H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 13

Drift change in bias with forecast lead-time τ ∂ D ( τ, X ) = b ( τ, X ) ∂τ , H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 14

Drift change in bias with forecast lead-time τ ∂ D ( τ, X ) = b ( τ, X ) ∂τ , H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 14

Quantifying drift � ( D ( τ, X )) 2 � D ( τ, X ) � = �� ∂ � � 2 � = b ( τ, X ) ∂τ Igor Kröner Drift Quantification and Correction in Decadal Predictions of Climate Extremes Indices , in preparation , H. Rust, FU Berlin, Drift in Decadal Prediction, 7th Int. Verification Methods Workshop, Berlin, May 11th, 2017 15