Can simple MOS bring improvement into ALADIN T 2m forecast? ak - - PowerPoint PPT Presentation

Can simple MOS bring improvement into ALADIN T2m forecast?

Ivan Baˇ st´ ak ˇ Dur´ an J´ an Maˇ sek Commenius University Slovak HydroMeteorological Institute 15th ALADIN Workshop Bratislava, 6.–10.6.2005

Why do we need model output statistics (MOS)?

• outputs from NWP models are not perfect, but are subject to errors
• these errors can be reduced by:
• 1. improving the numerical model (preferred way)
• 2. statistical adaptation of model outputs against observations
• first approach removes the source of errors,

but it is slower, expensive and requires joint effort of big teams

• second approach views model as black box, it can be implemented

quickly and cheaply, but the black box should not change

• with second approach we can hope to eliminate systematic part
• f model errors

2

What is MOS?

• MOS = multilinear regression:

Y =

m

• i=1

bi Xi

• ˆ

Y

+ ε Y – predictant (observed quantity) ˆ Y – MOS estimate of Y b1, . . . , bm – regression coefficients X1, . . . , Xm – predictors (quantities forecasted by model,

• bservations available at analysis time, . . . )

ε – error of MOS estimate

• regression coefficients are determined by least squares method, i.e.

by minimization of mean square error (ˆ Y − Y )2 on training data set

• MOS skill is evaluated using independent testing data set

3

MOS limitations

• number of predictors must be much smaller than size of training

data set (selection of too many predictors leads to overfitting)

• training period should be sufficiently long (in ideal case 5 years or

more) in order to correctly sample different weather situations

• time series of model outputs should be homogeneous (numerical

model should not change during period of MOS training and usage)

4

Questions to be answered

• 1. Can simple MOS improve ALADIN T2m forecast despite frequent

model changes?

• 2. What would be optimal design of the MOS system?
• 3. Can

more sophisticated MOS bring substantial improvement compared to simple MOS?

5

Used data

• studied period: 2000–2004 (5 years)
• observations:

SYNOP T2m observations from 9 Slovak stations

• forecasts:

ALADIN pseudoTEMPs (forecasted vertical profiles of pressure, temperature, humidity and wind) – used operational models: Jan 2000 – Dec 2002 ALADIN/LACE Prague Jan 2003 – Jun 2004 ALADIN/LACE Vienna Jul 2004 – Dec 2004 ALADIN/SHMU Bratislava – restriction to 00 UTC integration – concentration on +36 h forecast

6

Selected stations:

1 7 ˚ 1 7 ˚ 18˚ 18˚ 19˚ 19˚ 20˚ 20˚ 21˚ 21˚ 22˚ 22˚ 48˚ 48˚ 49˚ 49˚ 11976 11841 11816 11978 11819 11927 11952 11856 11867 100 200 400 600 1000 1500 2000 2500 7

Autocorrelation function of model T2m error (forecast against analysis):

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

correlation coefficient

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

correlation coefficient

3 6 9 12 15 18 21 24 27 30 33 36 39 42 45 48

forecast range [h]

(period 2000–2004, 00 UTC integration, average over all stations)

8

Evolution of T2m BIAS:

• 3
• 2
• 1

1 2 3

BIAS [oC]

• 3
• 2
• 1

1 2 3

BIAS [oC]

2000 2001 2002 2003 2004

smoothing window 30 days smoothing window 365 days

(00 UTC integration, +36 h forecast, all stations)

9

Design of simple MOS

• separate regression model for each station
• predictant: error of model T2m forecast (T +

F − T + O )

• predictors: 1, error of model T2m analysis (T 0

F − T 0 O), cos θ, sin θ,

cos 2θ, sin 2θ; where θ is time of year (goes from 0 to 2π)

• time predictors cos θ, sin θ, cos 2θ and sin 2θ are included in order to

describe annual course of model BIAS

• alternative way is to cluster data into several groups according to

part of year and develop separate MOS for each group: training testing 2000 2001 2002 2003

• 10

Annual course of T2m BIAS: +24 h forecast:

• 0.85
• 0.80
• 0.75
• 0.70
• 0.65
• 0.60
• 0.55
• 0.50

BIAS [oC]

• 0.85
• 0.80
• 0.75
• 0.70
• 0.65
• 0.60
• 0.55
• 0.50

BIAS [oC]

I II III IV V VI VII VIII IX X XI XII

+36 h forecast:

0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75

BIAS [oC]

0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75

BIAS [oC]

I II III IV V VI VII VIII IX X XI XII

(period 2000–2004, 00 UTC integration, all stations)

11

Tested configurations

• predictor selections:

p1a . . . 1 (simple BIAS correction) p2a . . . 1, T 0

F − T 0 O

p4a . . . 1, T 0

F − T 0 O, cos θ, sin θ

p6a . . . 1, T 0

F − T 0 O, cos θ, sin θ, cos 2θ, sin 2θ

• time window for data clustering: 1, 2, 3, 6 and 12 months

(12 months means no clustering)

• training period: 1, 2 and 3 years
• testing period: 1 year

12

T2m RMSE reduction, testing year 2003:

5 10 15 20

RMSE reduction [%]

5 10 15 20

RMSE reduction [%]

p1a p2a p4a p6a

1 2 3 1 2 3 1 2 3 1 2 3

1 2 3 6 12 1 2 3 6 12 1 2 3 6 12 1 2 3 6 12 1 2 3 6 12 1 2 3 6 12 1 2 3 6 12 1 2 3 6 12 1 2 3 6 12 1 2 3 6 12 1 2 3 6 12 1 2 3 6 12

2002-2003 2001-2003 2000-2003

(00 UTC integration, +36 h forecast, all stations)

13

T2m error distribution for station 11841 ˇ Zilina, testing year 2003:

5 10 15 20

frequency [%]

5 10 15 20

frequency [%]

• 12 -11 -10 -9
• 8
• 7
• 6
• 5
• 4
• 3
• 2
• 1

1 2 3 4 5 6 7 8 9 10 11 12

T2m error [oC]

• 12 -11 -10 -9
• 8
• 7
• 6
• 5
• 4
• 3
• 2
• 1

1 2 3 4 5 6 7 8 9 10 11 12

T2m error [oC] DMO MOS

(predictor selection p6a, training period 2000-2002, time window 12 months, 00 UTC integration, +36 h forecast)

14

T2m RMSE for individual stations, testing year 2003:

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0

RMSE of T2m [oC]

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0

RMSE of T2m [oC] station 1 1 8 1 6 1 1 8 1 9 1 1 8 4 1 1 1 8 5 6 1 1 8 6 7 1 1 9 2 7 1 1 9 5 2 1 1 9 7 6 1 1 9 7 8 DMO MOS

(predictor selection p6a, training period 2000-2002, time window 12 months, 00 UTC integration, +36 h forecast)

15

T2m BIAS for individual stations, testing year 2003:

• 4.5
• 4.0
• 3.5
• 3.0
• 2.5
• 2.0
• 1.5
• 1.0
• 0.5

0.0 0.5 1.0

BIAS of T2m [oC]

• 4.5
• 4.0
• 3.5
• 3.0
• 2.5
• 2.0
• 1.5
• 1.0
• 0.5

0.0 0.5 1.0

BIAS of T2m [oC] station 1 1 8 1 6 1 1 8 1 9 1 1 8 4 1 1 1 8 5 6 1 1 8 6 7 1 1 9 2 7 1 1 9 5 2 1 1 9 7 6 1 1 9 7 8 DMO MOS

(predictor selection p6a, training period 2000-2002, time window 12 months, 00 UTC integration, +36 h forecast)

16

T2m SDEV for individual stations, testing year 2003:

0.0 0.5 1.0 1.5 2.0 2.5 3.0

SDEV of T2m [oC]

0.0 0.5 1.0 1.5 2.0 2.5 3.0

SDEV of T2m [oC] station 1 1 8 1 6 1 1 8 1 9 1 1 8 4 1 1 1 8 5 6 1 1 8 6 7 1 1 9 2 7 1 1 9 5 2 1 1 9 7 6 1 1 9 7 8 DMO MOS

(predictor selection p6a, training period 2000-2002, time window 12 months, 00 UTC integration, +36 h forecast)

17

Results from simple MOS I

• the longer training period, the better MOS results
• including of analysis error among predictors improves MOS skill

compared to simple BIAS correction

• data clustering or use of time predictors improves MOS performance
• combination of data clustering with use of time predictors leads

to overfitting especially for short time windows and short training periods

• RMSE reduction is achieved by correcting yearly BIAS
• for best configurations overall RMSE reduction reaches 18%
• the most attractive candidate seems to be:

p6a, training period 3 years, time window 12 months

18

T2m RMSE reduction, testing year 2004:

5 10 15 20

RMSE reduction [%]

5 10 15 20

RMSE reduction [%]

p1a p2a p4a p6a

1 2 3 1 2 3 1 2 3 1 2 3

1 2 3 6 12 1 2 3 6 12 1 2 3 6 12 1 2 3 6 12 1 2 3 6 12 1 2 3 6 12 1 2 3 6 12 1 2 3 6 12 1 2 3 6 12 1 2 3 6 12 1 2 3 6 12 1 2 3 6 12

2003-2004 2002-2004 2001-2004

(00 UTC integration, +36 h forecast)

19

Results from simple MOS II

• results from testing year 2004 are bad surprise
• previously selected optimal configuration reaches overall RMSE

reduction only 7%

• longer training period does not imply better MOS performance
• data clustering or use of time predictors deteriorate MOS results
• best configuration is now: p2a, training period 2 years, time window

12 months; with overall RMSE reduction 9%

20

Cause of simple MOS failure

• during period 2000–2003 there were many changes in operational

model ALADIN: al11 → al12op3 → al15 → al25t2 different physical parametrisations and their tunings (CYCORA, CYCORA bis, CYCORA ter+++) dynamical adaptation, blending 6 h → 3 h coupling frequency, 31 → 37 vertical levels

• however, there was no change in horizontal geometry
• in 2004, model resolution changed from 12.2 km to 9.0 km
• it seems that related change of model orography is a critical factor

for behaviour of T2m error

• this is not surprising, since there is significant altitude dependence
• f T2m

21

Design of more sophisticated MOS

• possibility of regionalized regression model (i.e. common regression

equation for all stations with geographical predictors added)

• predictant: observed T2m at forecast time
• potential predictors:

1, observed T2m at analysis time forecasted quantities p, T, r, u, v and

• u2 + v2 at heights 2, 20,

200 and 2000 m above model surface time predictors: cos θ, sin θ, cos 2θ, sin 2θ geographical predictors: λ, ϕ, hmodel − h

• possibility of data clustering (less attractive alternative to the use
• f time predictors, since it reduces size of training data set)
• selection of final predictors by forward screening

22

T2m error distribution for station 11841 ˇ Zilina, testing year 2004: individual MOS equations: regionalized MOS equation:

5 10 15 20

frequency [%]

5 10 15 20

frequency [%]

• 12 -11 -10 -9
• 8
• 7
• 6
• 5
• 4
• 3
• 2
• 1

1 2 3 4 5 6 7 8 9 10 11 12

T2m error [oC]

• 12 -11 -10 -9
• 8
• 7
• 6
• 5
• 4
• 3
• 2
• 1

1 2 3 4 5 6 7 8 9 10 11 12

T2m error [oC] DMO MOS

5 10 15 20

frequency [%]

5 10 15 20

frequency [%]

• 12 -11 -10 -9
• 8
• 7
• 6
• 5
• 4
• 3
• 2
• 1

1 2 3 4 5 6 7 8 9 10 11 12

T2m error [oC]

• 12 -11 -10 -9
• 8
• 7
• 6
• 5
• 4
• 3
• 2
• 1

1 2 3 4 5 6 7 8 9 10 11 12

T2m error [oC] DMO MOS

(training period 2001–2003, time window 3 months, no time predictors, 00 UTC integration, +36 h forecast)

23

T2m RMSE for individual stations, testing year 2004: individual MOS equations: regionalized MOS equation:

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0

RMSE of T2m [oC]

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0

RMSE of T2m [oC] station 11816 11819 11841 11856 11867 11927 11952 11976 11978 DMO MOS

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0

RMSE of T2m [oC]

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0

RMSE of T2m [oC] station 11816 11819 11841 11856 11867 11927 11952 11976 11978 DMO MOS

(training period 2001–2003, time window 3 months, no time predictors, 00 UTC integration, +36 h forecast)

24

T2m BIAS for individual stations, testing year 2004: individual MOS equations: regionalized MOS equation:

• 3.0
• 2.5
• 2.0
• 1.5
• 1.0
• 0.5

0.0 0.5 1.0 1.5 2.0

BIAS of T2m [oC]

• 3.0
• 2.5
• 2.0
• 1.5
• 1.0
• 0.5

0.0 0.5 1.0 1.5 2.0

BIAS of T2m [oC] station 1 1 8 1 6 1 1 8 1 9 1 1 8 4 1 1 1 8 5 6 1 1 8 6 7 1 1 9 2 7 1 1 9 5 2 1 1 9 7 6 1 1 9 7 8 DMO MOS

• 3.0
• 2.5
• 2.0
• 1.5
• 1.0
• 0.5

0.0 0.5 1.0 1.5 2.0

BIAS of T2m [oC]

• 3.0
• 2.5
• 2.0
• 1.5
• 1.0
• 0.5

0.0 0.5 1.0 1.5 2.0

BIAS of T2m [oC] station 1 1 8 1 6 1 1 8 1 9 1 1 8 4 1 1 1 8 5 6 1 1 8 6 7 1 1 9 2 7 1 1 9 5 2 1 1 9 7 6 1 1 9 7 8 DMO MOS

(training period 2001–2003, time window 3 months, no time predictors, 00 UTC integration, +36 h forecast)

25

Results from more sophisticated MOS

• when model orography does not change, individual MOS equations

give better results than regionalized MOS (not shown)

• individual MOS equations are not usable when model orography

changes (for shown configuration overall RMSE increased by 13%, results are even worse than for simple MOS)

• however, regionalized MOS equation is usable despite the change of

model orography (for shown configuration overall RMSE decreased by 15%)

• regionalized MOS equation reduces RMSE mainly by correcting

yearly BIAS

• there is slight reduction of yearly SDEV, probably thanks to fitting

the annual course of model BIAS (not shown)

26

Conclusions

• at current horizontal resolutions (≈ 10 km), MOS still can improve

T2m forecast

• strongest limitation does not come from modifications of physical

parametrizations, but from changes of model orography

• in order to cope with this problem, regionalized approach must

be used, including hmodel − h among predictors

• care must be taken to selection of predictors, since overfitting

can occur even if only few predictors used

• preferable way how to describe annual course of model BIAS is to

use time predictors

• except from effect of regionalization, more sophisticated MOS does

not bring much improvement compared to simple MOS

27