The 2012 Moody's Analytics Risk Practitioner Conference The Validity of Validation Ottmar Bongers Chicago, 17 th Oct. 2012 | 06.10.2012 | Seite 1
List of Disclaimer 1. All characters/financial institutions in this presentation are fictitious, any resemblance to real persons/financial institutions, living or dead is purely coincidental. 2. All statements in this presentation do not represent BaFin but are just the presenter’s personal view. | 06.10.2012 | Seite 2
The Validity of Validation 1. Model Risk Management in the Basel III World 2. Dimensions of Model Validation 3. Validation of Economic Capital 4. Senior Management´s Role in Validation 5. Use of Validation in Model Supervision 6. Conclusion | 06.10.2012 | Seite 3
1. Model Risk Management in the Basel III World lessons learned ? Models in the financial crisis Pricing: overreliance on market liquidity / ratings • enormous uncertainty Market Risk: poor segmentation, risk factors, mapping • e.g. ABS, CDOs Capital: underestimation of required capital • insufficient risk analysis of products & markets insufficient awareness about model weaknesses & limitations insufficient speed of model adjustments not really a good track record | 06.10.2012 | Seite 4
1. Model Risk Management in the Basel III World Regulatory Changes for Internal Models (Pillar 1) structured products banished capital increase for market risk CRM for correlation trading by factor X IRC for migration risk, end of „surcharge“ Stress-VaR in memoriam financial crisis higher capital ratios CVA on the horizon More Internal Models, more capital, more need for validation! | 06.10.2012 | Seite 5
1. Model Risk Management in the Basel III World Changes in the Industry on Models ? But for how long ? some banks are gone, some merged some have been rescued by the tax payer capital programs driven by regulatory stress tests and ratios ICAAP & stress tests are taken more serious More model use, more challenge, more need for validation! | 06.10.2012 | Seite 6
The Validity of Validation 1. Model Risk Management in the Basel III World 2. Dimensions of Model Validation 2.1 Scope & Objectives of Model Validation 2.2 Validation Process 2.3 Documentation – Report - Action 3. Validation of Economic Capital 4. Senior Management´s Role in Validation 5. Use of Validation in Model Supervision 6. Conclusion | 06.10.2012 | Seite 7
2.1 Scope & Objectives of Model Validation Model Validation is … Fit for purpose? … set of processes and activities … to verify that models are performing as expected, in line with their design objectives and business uses … to identify potential limitations and assumptions and access their possible impact * * FED/OCC, Supervisory Guidance on Model Risk Management, 4.4.2011 | 06.10.2012 | Seite 8
2.1 Scope & Objectives of Model Validation Objects of Model Validation EC enterprise risk- and capital models regulatory capital VaR/ST per silo: risk models per risk type MR CR OpR BR LR.. parameter estimates (e.g. IRB) valuation models pricing + basis for risk models | 06.10.2012 | Seite 9
2.1 Scope & Objectives of Model Validation model exp. design data parameter exposure R data exp. I data F S proxies O risk measures K R E VaR M risk drivers C Stress Test O A D S E T historic/market L data risk model: much more than a formula | 06.10.2012 | Seite 10
2.1 Scope of Model Validation risk analysis / model coverage model design model input : parameter/assumptions/ C omprehensive exposure data / market data V alidation processing reporting use of the model results | 06.10.2012 | Seite 11
2.1 Scope of Model Validation Objectives of comprehensive model validation: • effective challenge • transparency of model strength & weaknesses • materiality of weaknesses • impact of weaknesses on model use • improvements of models and their use efficient model based risk/capital management processes | 06.10.2012 | Seite 12
2.2 Model Validation Process risk analysis/model coverage per risk type/per business lines only validation of model design: for model development only for model audit only model input: no periodic review no data quality check processing: no quality metric no measurement of process quality model results: no assessment of model risk no escalation threshold Limited validity of validation ! | 06.10.2012 | Seite 13
2.2 Model Validation Process risk analysis/model coverage: established periodic process per risk type and across silos/business lines: assessment of changes in products, clients, exposures, risk profile, market conditions validation of model design: assessment of design/parameter/assumptions model input: based on results of risk analysis and model use tolerance level for data and process quality processing: measurement of key process indicators model results: assessment of model risk well defined level of comfort for model risk escalation thresholds | 06.10.2012 | Seite 14
2.3 Documentation – Report - Action documentation: lack of governance, policies and procedures lack of documented validation approaches lack of documented validation results validation report: tons of statistical tests information overkill no assessment of materiality no assessment of impact on model use no recommendations action: none, at least not triggered by report Validation process neither efficient nor reliable for senior management nor auditable for internal audit! | 06.10.2012 | Seite 15
2.3 Documentation – Report - Action Documentation governance with roles & responsibilities policy with validation objectives and transparent model risk appetite procedures describing policy´s implementation per validation: approach and results | 06.10.2012 | Seite 16
2.3 Documentation – Report - Action Validation Report model weaknesses and limitations: materiality and impact on use of model results compensation measures monitoring of risk tolerance thresholds tracking of remediation measures recommendations for “model” improvements actionable executive summary Efficient & documented processes & actionable reports are a prerequisite for the validity of a C omprehensive V alidation ! | 06.10.2012 | Seite 17
The Validity of Validation 1. Model Risk Management in the Basel III World 2. Dimensions of Model Validation 3. Validation of Economic Capital 3.1 Objectives, Roles & Responsibilities 3.2 Aspects of EC Validation 3.3 Documentation – Report - Action 4. Senior Management´s Role in Validation 5. Use of Validation in Model Supervision 6. Conclusion | 06.10.2012 | Seite 18
3.1 Objectives, Roles & Responsibilities EC-Model fit for purpose ? What purpose? • ICAAP Capital Adequacy Assessment Process - EC < risk bearing capacity - capital planning / management • ERM Enterprise Risk Management - risk appetite - risk limits across risk types • VBM Value Based Management - resource allocation - risk return steering Without objective no validation ! | 06.10.2012 | Seite 19
3.1 Objectives, Roles & Responsibilities • coop Quants/Risk Managers • validation only by Quants • input data, model, results • Business “sign off” not transparent • quality challenge • no “buy in” of Business • acceptance, use • Board core “EC - User” • no real use of EC • no “client” – no budget - • Risk-/Capital Committees no improvements - no quality • approval on material changes | 06.10.2012 | Seite 20
3.1 Objectives, Roles & Responsibilities • mandated owner of EC validation process • validation team or “independence” by segregation of duties • leverage on risk silo validation “effective challenge” is key criteria cross silo risk analysis / risk management • mandated committees / ultimate responsibility: Board • dedicated experienced teams • periodic/triggered ad hoc review of risk map Without “clients”, mandate, resources and effective challenge no validity of validation ! | 06.10.2012 | Seite 21
The Validity of Validation 1. Model Risk Management in the Basel III World 2. Dimensions of Model Validation 3. Validation of Economic Capital 3.1 Objectives, Roles & Responsibilities 3.2 Aspects of EC Validation 3.3 Documentation – Report - Action 4. Senior Management´s Role in Validation 5. Use of Validation in Model Supervision 6. Conclusion | 06.10.2012 | Seite 22
3.2 Aspects of EC Validation: Credit Risk risk analysis: products, markets, clients model design/ e.g. credit spread – migration – default risk model coverage: trading/banking book in line with use: going/gone concern market data: e.g. migration matrices parameter: IRB parameter PD, LGD, CCF recalibration asset correlation, R² exposure: e.g. industry mapping, CCR model based EaD * * BCBS, Sound practices for backtesting counterparty credit risk models, 12/2010 | 06.10.2012 | Seite 23
Recommend
More recommend
