Towards a Unified Statistical Framework to Evaluate Financial Crises - PowerPoint PPT Presentation

Towards a Unified Statistical Framework to Evaluate Financial Crises Early Warning Systems How to evaluate an EWS? Bertrand Candelon † , Elena-Ivona Dumitrescu ‡ , Christophe Hurlin ‡ † Maastricht University and ‡ University of Orléans 2009

Introduction A New EWS Evaluation Method EWS Specification and Estimation Empirical Results Conclusions Introduction → From the subprime crisis to currency crises → Early Warning Systems (EWS) set up to ring before the occurence of crises Towards a Unified Statistical Framework to Evaluate Financial Crises Early Warning Systems 2 / 36

Introduction A New EWS Evaluation Method EWS Specification and Estimation Empirical Results Conclusions Introduction How can we specify an EWS model? → Rich literature (Kaminski et al. (1998), Kumar et al. (2003), Abiad (2003), etc.) How can we evaluate the predictive abilities of an EWS? → Kaminski et al. (1998) : signalling approach ◮ Threshold which minimizes the NSR criteria ◮ Type I and type II errors → Arbitrarely chosen cut-offs (Berg and Patillo (1999), Arias and Erlandsson (2005)) Towards a Unified Statistical Framework to Evaluate Financial Crises Early Warning Systems 3 / 36

Introduction A New EWS Evaluation Method EWS Specification and Estimation Empirical Results Conclusions Originality Our New EWS Evaluation Method → I. Optimal cut-off → II. Credit-scoring evaluation criteria QPS, LPS, AUC, Pietra Index, Bayesian Error, Kuiper’s score → III. Comparison tests ◮ Diebold-Mariano (1995) test for non-nested models ◮ Clark-West (2007) test for nested models ◮ Area under ROC comparison test Towards a Unified Statistical Framework to Evaluate Financial Crises Early Warning Systems 4 / 36

Introduction A New EWS Evaluation Method EWS Specification and Estimation Empirical Results Conclusions Contents A New EWS Evaluation Method EWS Specification and Estimation Empirical Results Conclusions Towards a Unified Statistical Framework to Evaluate Financial Crises Early Warning Systems 5 / 36

Introduction A New EWS Evaluation Method Optimal cut-off identification EWS Specification and Estimation Performance assessment criteria Empirical Results Comparison tests Conclusions Step 1. A New EWS Evaluation Method Towards a Unified Statistical Framework to Evaluate Financial Crises Early Warning Systems 6 / 36

Introduction A New EWS Evaluation Method Optimal cut-off identification EWS Specification and Estimation Performance assessment criteria Empirical Results Comparison tests Conclusions I. Optimal cut-off identification C ∗ = Arg { C } [ Sensitivity ( C ) = Specificity ( C )] , where C ∈ [ 0 , 1 ] Definition 1. Sensitivity is the number of crises correctly predicted for a cutoff C over the total number of crises in the sample Definition 2. 1 − Specificity is the number of false alarms for a cutoff C over the total number of non-crises in the sample Towards a Unified Statistical Framework to Evaluate Financial Crises Early Warning Systems 7 / 36

Introduction A New EWS Evaluation Method Optimal cut-off identification EWS Specification and Estimation Performance assessment criteria Empirical Results Comparison tests Conclusions Towards a Unified Statistical Framework to Evaluate Financial Crises Early Warning Systems 8 / 36

Introduction A New EWS Evaluation Method Optimal cut-off identification EWS Specification and Estimation Performance assessment criteria Empirical Results Comparison tests Conclusions II. Performance assessment criteria The Area Under the ROC Curve and the Quadratic Probability Score What is the ROC curve? (Receiving Operating Characteristic) Towards a Unified Statistical Framework to Evaluate Financial Crises Early Warning Systems 9 / 36

Introduction A New EWS Evaluation Method Optimal cut-off identification EWS Specification and Estimation Performance assessment criteria Empirical Results Comparison tests Conclusions II. Performance assessment criteria The Area Under the ROC Curve � 1 A = Sensitivity ( 1 − Specificity ) d ( 1 − Specificity ) 0 ◮ Measure of the model’s overall ability to discriminate between the cases correctly predicted and the false alarms ◮ For a perfect model AUC=1 while for a random one AUC=0.5 Towards a Unified Statistical Framework to Evaluate Financial Crises Early Warning Systems 10 / 36

Introduction A New EWS Evaluation Method Optimal cut-off identification EWS Specification and Estimation Performance assessment criteria Empirical Results Comparison tests Conclusions II. Performance assessment criteria The Quadratic Probability Score T � QPS = 1 2 ( � I t − I t ) 2 T t = 1 ◮ Comparison of forecasts ( � I t ) and realizations ( I t ) ◮ The closer QPS is to 0 the better the model is Towards a Unified Statistical Framework to Evaluate Financial Crises Early Warning Systems 11 / 36

Introduction A New EWS Evaluation Method Optimal cut-off identification EWS Specification and Estimation Performance assessment criteria Empirical Results Comparison tests Conclusions III. Comparison tests 1. Diebold-Mariano (1995) test for non-nested models 2. Clark-West (2007) test for nested models 3. Area under ROC comparison test (Delong et al. (1988)) Towards a Unified Statistical Framework to Evaluate Financial Crises Early Warning Systems 12 / 36

Introduction A New EWS Evaluation Method Optimal cut-off identification EWS Specification and Estimation Performance assessment criteria Empirical Results Comparison tests Conclusions III. Comparison tests Proposition 1 : Let us denote by M 1 and M 2 two EWS models, and by � AUC 1 and � AUC 2 the associated areas under the ROC curve. H 0 : � AUC 1 = � AUC 2 ( � AUC 1 − � AUC 2 ) 2 d T →∞ χ 2 ( 1 ) − − − − → Var ( � AUC 1 − � AUC 2 ) Towards a Unified Statistical Framework to Evaluate Financial Crises Early Warning Systems 13 / 36

Introduction A New EWS Evaluation Method Currency crisis dating method EWS Specification and Estimation Empirical models Empirical Results Conclusions Step 2. EWS Specification and Estimation Towards a Unified Statistical Framework to Evaluate Financial Crises Early Warning Systems 14 / 36

Introduction A New EWS Evaluation Method Currency crisis dating method EWS Specification and Estimation Empirical models Empirical Results Conclusions To apply our evaluation methodology: I. Real crisis dating method ( I t ) → KLR modified pressure index - Lestano and Jacobs (2004) → The threshold equals two standard deviations above the mean II. Crisis probabilities ( � Pr t ) → Panel logit with fixed effects → Markov Switching Model with constant transition probabilities Towards a Unified Statistical Framework to Evaluate Financial Crises Early Warning Systems 15 / 36

Introduction A New EWS Evaluation Method Currency crisis dating method EWS Specification and Estimation Empirical models Empirical Results Conclusions I. Currency crisis dating method KLR modified pressure index - Lestano and Jacobs (2004) Definition 3. The 24 months crisis variable:  24 �   1 , if Crisis n , t + j > 0 I t = C 24 n , t = j = 1   0 , otherwise Towards a Unified Statistical Framework to Evaluate Financial Crises Early Warning Systems 16 / 36

Introduction A New EWS Evaluation Method Currency crisis dating method EWS Specification and Estimation Empirical models Empirical Results Conclusions II. Empirical models Model 1. Panel and time-series logit model ′ x + f n ) exp ( β Pr ( C 24 nt = 1 ) = ′ x + f n ) ∀ n ∈ Ω h , 1 + exp ( β where ◮ f n represents the fixed effects ◮ x is the matrix of economic variables ◮ n is the country identifier ◮ Ω h is the h th cluster Optimal country clusters: (Kapetanios procedure (2003)) Towards a Unified Statistical Framework to Evaluate Financial Crises Early Warning Systems 17 / 36

Introduction A New EWS Evaluation Method Currency crisis dating method EWS Specification and Estimation Empirical models Empirical Results Conclusions II. Empirical models Model 2. Markov model - Hamilton (1995) KLRm t = µ t ( S t ) + β ( S t ) x t + ǫ t ( S t ) , where ◮ KLRm t is the pressure index vector ◮ x t represents the matrix of economic variables ◮ S t follows a two states Markov chain � 1 , if there is a crisis at time t S t = 0 , if not Towards a Unified Statistical Framework to Evaluate Financial Crises Early Warning Systems 18 / 36

Introduction A New EWS Evaluation Method Currency crisis dating method EWS Specification and Estimation Empirical models Empirical Results Conclusions II. Empirical models Definition 4. The 24 months ahead forecasts (Arias and Erlandson (2005)): Pr ( S t + 1 ... t + 24 = 1 | Ω t ) = 1 − Pr ( S t + 1 ... t + 24 = 0 | Ω t ) = 1 − { [ P 10 P ( 23 ) 00 Pr ( S t = 1 | Ω t )] + [ P 24 00 Pr ( S t = 0 | Ω t )] } , ◮ where P 10 and P 00 are elements of the transition probability matrix Towards a Unified Statistical Framework to Evaluate Financial Crises Early Warning Systems 19 / 36

Introduction A New EWS Evaluation Method Currency crisis dating method EWS Specification and Estimation Empirical models Empirical Results Conclusions II. Empirical models From crisis probabilities to crisis forecasts � 1 , if Pr ( C 24 t = 1 ) > C ∗ � I t = , 0 , otherwise where C ∗ is an optimal cut-off (see section 1) Towards a Unified Statistical Framework to Evaluate Financial Crises Early Warning Systems 20 / 36