Empirical Simultaneous Confidence Regions for Path-Forecasts Òscar Jordà 1 , Malte Knüppel 2 , Massimiliano Marcellino 3 1 University of California, Davis 2 Deutsche Bundesbank 3 European University Institute, Università Bocconi and CEPR August 2010 Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 1 / 18
Motivation Confidence bands are used by many forecasting institutions which publish multi-period-ahead forecasts (pathcasts) These confidence bands contain intervals where the realizations are going to lie in with a certain probability for each single period However, path forecasts are often more important than forecast for a single period “Deflation is a decline in the general level of prices... It has to be persistent — and last for an extended period of time, ...” (Bini Smaghi, ECB Board member, 2009) “The entire path for inflation and the real economy, both before and after the two years, will be taken into account when setting interest rates.“ (Jarle Bergo, Deputy Governor, Norges Bank, 2004) Pricing path-dependent “exotic” options In such cases, confidence bands should be calculated for path forecasts, not for single h -step-ahead forecast Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 2 / 18
Motivation Confidence regions for impulse responses developed by Jorda (2009, REStat) Methodology applied to path forecasts by Jorda & Marcellino (2010, JAE) Jorda & Marcellino study case of known forecasting model, derive model-based confidence regions In most central banks (Fed, ECB, Bank of England, Sveriges Riksbank, Deutsche Bundesbank...) it is standard practice to determine forecast uncertainty based on past forecast errors Aim of this work: Investigate properties of confidence regions for path forecasts if forecasting model is unknown, but sample of forecast errors is available Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 3 / 18
Introduction - Path Forecast Uncertainty Fan charts convey information about forecast uncertainty several periods ahead But confidence intervals are marginal bands for single periods, do not contain information about probability of paths Bundesbank Bank of England Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 4 / 18
Introduction - An Example Suppose y t = ρ y t − 1 + ε t , ε t ∼ N ( 0 , 1 ) Then 1- and 2-step-ahead forecast errors are distributed as � u t + 1 � �� 0 � � 1 �� ρ ∼ N , 1 + ρ 2 u t + 2 0 ρ Forecast errors have a joint distribution, thus forecast paths can be ranked according to their p -values 4 4 upper marginal upper marginal lower marginal lower marginal ρ = 0 . 9, 95% marginal path 1 path 1 3 3 path 2 path 2 path 3 path 3 confidence bands 2 2 path 4 path 4 path 5 path 5 path 6 path 6 1 1 path 1: p -value 0.06 y(t) y(t) 0 0 path 2: p -value 0.00 -1 -1 path 3: p -value 0.08 -2 -2 paths 4,5,6: p -value 0.05 -3 -3 bands: p -value 0.10 -4 -4 0 0 0.2 0.2 0.4 0.4 0.6 0.6 0.8 0.8 1 1 1.2 1.2 1.4 1.4 1.6 1.6 1.8 1.8 2 2 forecast horizon forecast horizon Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 5 / 18
Introduction - Example Cont’d p -values correspond to multiple testing procedure with H 0 : u t + 1 = 0 � u t + 2 = 0, resulting in a confidence ellipse . Looking at the forecast error distribution from above ( ρ = 0 . 9): 3 2 1 95% conf. ellipse u(t+2) 0 2 s.e. box Scheffé box -1 -2 -3 -2 0 2 u(t+1) Drawing confidence bands produces rectangular regions ⇒ Confidence ellipse cannot be represented in fan charts. But rectangular region can at least be chosen such that confidence bands have correct “Wald coverage” Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 6 / 18
Introduction - Scheffé Bands Correct "Wald coverage" means: We want 100 · α % of all paths to have a lower p -value than the confidence bands Scheffé bands proposed by Jorda (2009) have correct Wald coverage (or FDR control = false discovery rate control), are easy to construct, have interesting statistical interpretation. But what about their coverage in the conventional sense? That is, what is the percentage of paths lying within the Scheffé bands given α ? Equivalent question: How good is their FWER control ( = family-wise error control)? Marginal bands fail at FWER control, dramatically so when H is large Another alternative: Bonferroni bands, based on inequality Pr ( A 1 ∪ A 2 ∪ . . . ∪ A H ) ≤ ∑ H i = 1 Pr ( A i ) , equivalent to marginal bands with critical value α H Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 7 / 18
Introduction - Example Cont’d � u t + 1 � �� 0 � � 1 �� ρ Forecast errors ∼ N , 1 + ρ 2 u t + 2 0 ρ Forecast path given by ˆ Y ( H ) Set α = 0 . 05 � � 1 � Marginal bands: ˆ Y ( H ) ± 1 . 96 1 + ρ 2 � � 1 � Bonferroni bands: ˆ Y ( H ) ± 2 . 24 1 + ρ 2 � 1 � � � 1 � 0 Scheffé bands: ˆ 5 . 99 Y ( H ) ± 2 1 1 ρ Note: Only Scheffé bands use entire covariance matrix, other bands only use diagonal elements Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 8 / 18
Introduction - Example Cont’d Confidence bands for H = 2 and H = 12 , α = 0 . 05 , ρ = 0 . 9 4 10 3 2 5 1 y(t) y(t) 0 0 -1 -2 -5 marginal marginal Bonferroni Bonferroni -3 Scheffé Scheffé -4 -10 0 0.5 1 1.5 2 0 2 4 6 8 10 12 forecast horizon forecast horizon Marginal bands are always narrower than Bonferroni bands Scheffé bands here are narrower than marginal bands at the beginning, wider than Bonferroni bands at the end Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 9 / 18
Introduction - Example Cont’d Coverage of bands, H = 2 , α = 0 . 32 0.75 0.9 nominal nominal marginal marginal 0.7 Bonferroni 0.8 Bonferroni Scheffé Scheffé 0.65 FWER control FDR control 0.7 0.6 0.6 0.55 0.5 0.5 0.45 0.4 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 ρ ρ Typical result: Bonferroni bands are too conservative, marginal bands have too small coverage Scheffé bands have exact Wald coverage and too small coverage with respect to FWER control Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 10 / 18
Empirical Confidence Regions Using past forecast errors to estimate variance-covariance matrix (vcv matrix) reasonable if forecasting model is misspecified forecasting model is not available Both reasons highly relevant in macroeconomics, probably therefore fan charts of most central banks based on past forecast errors Potential problem of Scheffé bands: Larger estimation uncertainty, because entire vcv matrix has to be estimated Marginal and Bonferroni bands only require estimation of diagonal elements of vcv matrix Coverage of bands investigated with MC simulations Set-up similar to Jorda & Marcellino, but with misspecified models Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 11 / 18
Simulation Design DGP is Stock-Watson (2001, JEP) VAR(4) with variables unemployment, inflation, and federal funds rate, normally distributed shocks Coverage investigated for model-based and forecast-error-based (i.e. empirical) confidence bands Misspecifications of forecasting models: dynamic (1 lag instead of 4) omitted variable (unemployment missing) break in coefficients of DGP break in vcv matrix of DGP both breaks mentioned in DGP (all breaks assumed to occur in 1985) Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 12 / 18
Results Marginal bands are too narrow Bonferroni bands have good FWER control, but are often too narrow wrt to FDR control Scheffé bands have good FDR and reasonable FWER control Empirical Scheffé bands work well even if number of empirical forecast errors q is small (unless H is very large) For example, with H = 8 and q = 40, coverage is still not too far from nominal level For most misspecifications, model-based and empirical bands have similar coverage But break in vcv matrix leads to very poor results of all model-based bands, while empirical Scheffé bands retain good coverage Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 13 / 18
Application We look at the Greenbook forecasts for US growth (annualized quarterly growth rates of GNP/GDP) and inflation (annualized quarterly growth rates of GNP/GDP deflator) Choosing H = 5 gives 119 path forecasts from 1974q2 to 2003q4 In addition to final release data, real-time data are used to calculate forecast errors Normality tests of forecast errors do not reject in most cases We construct empirical bands based on a rolling window of forecast errors for 40 path forecasts This yields bands for 75 path forecasts with the first path starting in 1985q2 For these bands and path forecasts, we calculate the coverage rates Jordà, Knüppel, Marcellino ( ) Confidence Regions for Path-Forecasts 08/10 14 / 18
Recommend
More recommend
