How can we best combine forecasts for added value? J. Broecker 1 , - PowerPoint PPT Presentation

Thorpex Montreal 2004 How can we best combine forecasts for added value? J. Broecker 1 , L. Clarke 1 , D. Kilminster 2 and L.A. Smith 1 , 2 1. Department of Statistics, London School of Economics, UK 2. Pembroke College, Oxford, UK

Thorpex Montreal 2004 How can we best combine forecasts for added value? How might we use TIGGE? J. Broecker 1 , L. Clarke 1 , D. Kilminster 2 and L.A. Smith 1 , 2 1. Department of Statistics, London School of Economics, UK 2. Pembroke College, Oxford, UK

Motivation 1 Forecast improvement can be achieved in two ways: Motivation Overview Combining Evaluation Example

Motivation 1 Forecast improvement can be achieved in two ways: Motivation Overview • improving the models (strategic) Combining Evaluation Example

Motivation 1 Forecast improvement can be achieved in two ways: Motivation Overview • improving the models (strategic) Combining • using the available information more effectively (tactical) Evaluation Example

Motivation 1 Forecast improvement can be achieved in two ways: Motivation Overview • improving the models (strategic) Combining • using the available information more effectively (tactical) Evaluation Example THORPEX : “THORPEX will develop, demonstrate and evaluate a multi-model, multi-analysis and multi-national ensemble prediction system, referred to as TIGGE.”

Overview 2 • Combining Simulations Motivation • Evaluation Overview – skill scores → Broecker – bootstrapping and meaningful skill comparison Combining • Example - combining ECMWF and NCEP Evaluation Example

Inputs 3 Motivation Climatological Distribution Overview Combining Evaluation Example

Inputs 3 Motivation Climatological Dressed Point Distribution Forecast Overview x Combining Evaluation Example Roulston & Smith, Tellus 55 2003 Raftery et al. Univ. Washington Dept. of Stat. Tech. Report 440 2003

Inputs 3 Motivation Climatological Dressed Point Ensemble Distribution Forecast Product Overview x x x x x x x x x x x Combining Evaluation Example Roulston & Smith, Tellus 55 2003 Raftery et al. Univ. Washington Dept. of Stat. Tech. Report 440 2003

Inputs 3 Motivation Climatological Dressed Point Ensemble Distribution Forecast Product Overview x x x x x x x x x x x Combining Evaluation Example Combined Forecast Roulston & Smith, Tellus 55 2003 Raftery et al. Univ. Washington Dept. of Stat. Tech. Report 440 2003

Combining: Skill Scores 4 The combination is based on the skill of the Motivation final forecast Overview Combining s = S ( f, o ) Evaluation forecast distribution f Example verifying observation o skill score S skill of forecast f s

Combining 5 One combination method is to take a weighted sum of the component distributions Motivation � f = α i f i Overview Combining Choose α i that maximises the skill score � s � = max �S ( f, o ) � Evaluation Example over a set of historical forecast-verification pairs Ignorance : s = − log p ( o ) Roulston & Smith, Monthly Weather Review 130

Evaluation:Bootstrapping 6 Motivation Overview Combining Skill Evaluation Example Model A Model B Model C Out of sample, particular location, particular lead time, particular target

Evaluation:Bootstrapping 6 Motivation Overview Combining Skill Evaluation Example Model A Model B Model C Out of sample, particular location, particular lead time, particular target

The Right Comparison 7 We do not want to compare the uncertainty in the average performance of two models. Motivation Overview We want the uncertainty in the comparative Combining performance of the models to each other. Evaluation Bootstrap the difference: � s A − s B � BS Not the difference of the bootstraps: � s A � BS − � s B � BS Example

Comparative Skill 8 Motivation Overview Comparitive Skill Combining Evaluation Example A vs B A vs C B vs C

Example 9 Predicting temperature at Heathrow. Motivation Using Overview • NCEP high resolution • NCEP ensemble • ECMWF high resolution Combining • ECMWF ensemble Evaluation Evaluating using Ignorance - out of sample Example

Example - one lead time 10 Motivation Overview Comparitive Skill Combining Evaluation Example Lead Time (Days) Combination vs NCEP Hi Res. IGN: smaller = good

Example - one lead time 10 Motivation Overview Comparitive Skill Combining Evaluation Example Lead Time (Days) Combination vs NCEP Hi Res. Combination vs NCEP Ens. IGN: smaller = good

Example - one lead time 10 Motivation Overview Comparitive Skill Combining Evaluation Example Lead Time (Days) Combination vs NCEP Hi Res. Combination vs NCEP Ens. Combination vs ECMWF Hi Res. IGN: smaller = good

Example - one lead time 10 Motivation Overview Comparitive Skill Combining Evaluation Example Lead Time (Days) Combination vs NCEP Hi Res. Combination vs NCEP Ens. Combination vs ECMWF Hi Res. IGN: smaller = good Combination vs ECMWF Ens.

Example - all lead times 11 Motivation Overview Comparitive Skill Combining Evaluation Example Lead Time (Days) Combination vs NCEP Hi Res. IGN: smaller = good

Example - all lead times 11 Motivation Overview Comparitive Skill Combining Evaluation Example Lead Time (Days) Combination vs NCEP Hi Res. Combination vs NCEP Ens. IGN: smaller = good

Example - all lead times 11 Motivation Overview Comparitive Skill Combining Evaluation Example Lead Time (Days) Combination vs NCEP Hi Res. Combination vs NCEP Ens. Combination vs ECMWF Hi Res. IGN: smaller = good

Example - all lead times 11 Motivation Overview Comparitive Skill Combining Evaluation Example Lead Time (Days) Combination vs NCEP Hi Res. Combination vs NCEP Ens. Combination vs ECMWF Hi Res. IGN: smaller = good Combination vs ECMWF Ens.

Summary 12 We have: Motivation • presented a user-orientated methodology for com- bining simulations • whatever combination method, the evaluation must Overview be robust • dressing method, combination method and size of Combining forecast-verifi cation archive affects performance • potential relevance to TIGGE, provides a frame- Evaluation work for allowing users to extract the forecast in- formation most relevant to them Example