Professional Documents
Culture Documents
INTRODUCTION
This scenario might sound familiar: on a few key individuals. Processes related Operational Risks
to model monitoring may be affected Lack of automation (for example, the
A bank uses over 50 analytical models
by a number of elements including: manual entry of SAS outcomes into
to support its underwriting, pricing and
finance functions. There are numerous Microsoft Excel/Microsoft PowerPoint),
Governance regular controls in code and of tracking
models in place to generate the probability
of default (PD), loss given default (LGD) A lack of policies around model risk logs, leading to a high error rate.
and exposure at default (EAD) metrics monitoring, or policies in place may
not be properly enforced. Lack of contingency plans lead to the risk
that serve as inputs to the banks capital of losing key historical facts if dedicated
computation process. No full model list for audit tracking personnel leave the firm and adequate logs
Model monitoring and tracking are purposes; or a list not regularly updated. are not in place.
performed by an understaffed analytics
Organization In todays financial institutions, analytical
team, using Microsoft Excel templates
and SAS or other tools that may have The organization finding itself in reactive models are high-value organizational and
been handed down for years. mode, often scrambling frantically to meet strategic assets. As models are needed to
internal and external deadlines. run the business and comply with regulations,
Reports for senior management are they must be effectively and efficiently
assembled manually, under pressure, using Timelines regularly affected by poor managed for optimal performance once
metrics and formats often not updated for capacity planning and inadequate in production.
long periods of time. contingency plans.
Poor governance and process in the
A regulatory review of any models used Processes and Procedures management of these models can expose
usually triggers a massive manual exercise No standards in place surrounding an organization to the risks of suboptimal
as reports and documents are created and the frequency of model monitoring; business decisions, regulatory fines and
compiled. There may be multiple rounds similar teams using different standards reputational damage.
of information exchange with the regulator, and procedures for model tracking
as internal reports do not address all templates or for performance metrics. As seen in Figure 1 below, a robust system
aspects of model performance. of ongoing model monitoring is a key
Monitoring Output Analysis component in the management of model risk.
The above describes the situation at
a medium-sized European bank, but is fairly Poorly performing models remaining
in production due to decision making From a broader perspective, the term model
common across the industry. refers to any approach that processes
affected by inconsistency in metrics,
frequency, lack of analysis of root causes, quantitative data as input, and provides
Model monitoringthe regular analytical
or by ineffective and poor commentaries a quantitative output. The definition of
review of model performancemay be loosely
on monitoring output. a model can prove contentious. Best practice
managed, with its effectiveness dependent
often sees a broader definition of a model,
with a sensible model monitoring standards
document in place. This should help ensure
Figure 1: Managing Model Risk that all of the models are noted on the master
list, but that there are appropriate levels
of monitoring put in place.
Managing Model Risk
While we discuss the measurement of credit
First Line of Defense Second Line of Defense Third Line of Defense risk, and therefore refer to scoring or rating
PD and LGD models, the best practices
Model Development On Going Model Top Management
Monitoring and Board Review to which we refer are applicable to any
Model Documentation type of quantitative model.
Independent Validation Model Use Risk Escalation
Model Implementation and Stress Testing
Model Risk Appetite Setting
Model Usage Regular Model
Review Process Model Risk Management
Framework
2
3
EFFECTIVE MODEL MONITORING:
KEY PRINCIPLES
Ongoing monitoring is essential Guidelines for establishing a model inventory
include:
to evaluate whether changes
in products, exposures, Segregate the inventory building exercise
by model category; for example segments
activities, customers or market may include:
conditions call for adjustment, Underwriting/Application Scoring Models
redevelopment, or replacement Account/Customer Behavior Scoring
of the model or to verify that any Models
4
Table 1: Enterprise Level Model Inventory
Component Description
Model Usage What life-cycle process, product and entity does the model impact.
Model Adjustments What adjustments (if any) are made to the model output before it is fit for purpose.
Materiality Portfolio EAD (amount and percentage) covered by each model, and EAD period
date and source. If EAD is not available, portfolio balance should be used and noted.
For different model types, alternative materiality measures may be used.
For example, application model materiality may be measured by projected
pipeline. This should be clearly laid out in the model monitoring standards.
Model Developer Work contact details for employees involved in model creation.
Model Approver Work contact details for key employees involved in model approval.
Model User Work contact details for key employees involved in model usage.
Model Maintenance Work contact details for key employees involved in model maintenance.
Current Model Risk Rating The current risk rating of the model (e.g. Red/Amber/Green).
Underwriting Systems
6
GOVERNANCE STRUCTURE also examine all processes involved. This
is a critical element; in an anecdotal case,
Critical components of a robust governance the model monitoring Microsoft Excel
structure around credit risk model monitoring spreadsheets used by a large Asian regional
include: bank were found to have several formula
errors. The spreadsheets had been used for
Independence of the model monitoring several years and were assumed to be correct.
team from the model development team;
Effective model audit processes and Senior management and board involvement
procedures; and in model governance may be the most
important element of all. This is essential
Engagement and involvement from senior to help ensure awareness and ownership
management. of models and related issues, appropriate
decision making in relation to potential
While the necessity for an independent business and regulatory impacts, and existence
model monitoring team may seem obvious, of appropriate incentive structures. The
in practice, modeling functions are often communication lines of a good governance
loosely structured, and independence may framework may be found in Figure 3.
exist only in theory.
It is important in our view to have
Ideally the organization should have a clear communication across all three lines of
separation among model developers/users defense. Figure 3 outlines these lines of
and validation functions. Incentive structures communications for the functions previously
should not discourage the escalation of model identified in Figure 1. This can help ensure
issues as appropriate, with a clearly defined that low level model issues can be actioned
escalation matrix. quickly, without senior management
involvement. However, having an escalation
A robust internal audit process is a key
channel to senior management can help
element of any model monitoring program.
raise major issues should action be required.
The audit function would typically audit all
Internal audit also has a very clear role to
stakeholdersincluding developers, users,
play in helping to provide assurance that all
and monitoring/validation teamsand would
processes have been carried out effectively.
Audit Findings
Board Risk Committee
Risk Committee
Model Developers
Internal Audit
Model Owners/Users
Model Validations/Monitoring
7
8
COMPREHENSIVE INDICATORS Table 2: Evaluating Credit Risk Model Performance
Change in Gini Coefficient A comparison of the models most recent Gini calculation with that observed
for the previous tracking period. A large decrease or increase would indicate
a need for further investigation.
9
SYSTEM STABILITY Table 4: Evaluating System Stability
10
ACTUAL VERSUS EXPECTED Table 6: Evaluating Model Calibration
OR CALIBRATION
Component Description
The calibration or actual versus expected
performance of a model refers to its ability Hosmer-Lemeshow (HL) The HL test quantifies if the model is a good fit given the current data.
or Chi-Square Test It compares observed versus predicted counts of outcome defaults in each
to yield an accurate point prediction for
rating grade or score decile.
the output variable. This is particularly
relevant for Regulatory Capital models where Quality of fit is thus summarized into a single statistic.
systematic over or under prediction may be Brier Score The Brier score measures the accuracy of probabilistic predictions, measuring
subject to regulatory scrutiny and overrides. the mean squared differences between predicted and actual outcomes.
Table 6 discusses some measures
The score is only suitable for models predicting a binary outcome.
of calibration observed in the industry
and within vendor tools. Calibration Curve This is a technique based on a confidence interval created around the
Shape Test model prediction.
The Basel III norms explicitly cite appropriate Depending on the number of instances where the actual outcome lies
calibration of the risk functions, which outside this confidence interval, the test is classified as a Red, Amber
convert loss estimates into regulatory or Green outcome.
capital requirements.3 This level of scrutiny The calibration curve plot accompanying the test provides a useful visual
can be expected to increase within this view of model calibration.
important aspect of model performance.
Source: Accenture, November 2014
DISTRIBUTION
Component Description
Models which produce large concentrations
in particular credit grades or scores can Herfindahl-Hirschman Index The HHI has long been used as a measure of market concentration,
(HHI) with a value of 0 indicating near-perfect competition and a value
be problematic, and undesirable from a
of 1 indicating a monopoly.
regulatory perspective. A large bank was
asked to redevelop several rating models The same measure may be computed for concentrations by score/credit
grade in practice, values of the HHI in excess of 0.25 may indicate a need
that assigned more than 20 percent of the
for further action.
concerned portfolio to one credit grade.
The concentration of the score can be
measured by the Herfindahl-Hirschman Source: Accenture, November 2014
Index, which is outlined in Table 7.
11
EFFECTIVE MANAGEMENT INFORMATION
SYSTEMS (MIS)
There are numerous metrics Coverage: Using the model inventory Provide an Overall Assessment: Depending
discussed earlier, we suggest that all of the on performance outcome, model monitoring
that may be used to set models which are used for credit risk rating is usually performed to lead to one of three
up a comprehensive model be monitored on a regular basis. This will help possible decisions:
the organization be aware of the portfolio
monitoring system, these coverage of its various models. Performance Action
should be incorporated into Model still fits for purpose. No change
Data/Formula Integrity and Validation:
a robust and timely MIS Model monitoring tools should be subject Model still provides Recalibration
program. Some industry best to independent validation for data satisfactory discriminatory
performance but does not
aggregation processes and formulae
practices observed for model prior to the reporting cycle (for example, yield accurate point prediction.
risk monitoring MIS are: on a monthly cycle for data and quarterly Fall in discriminatory power, In-depth analysis
for formulae). Formula errors have been possibly due to a large of root causes and
Be Comprehensive: Often model monitoring observed in the model monitoring tools change in population. optional solutions.
systems within institutions present only the of at least one large European bank, leading Model still provides Decision to be taken
KS and Gini statistics, which may mask serious to a regulatory reprimand. satisfactory discriminatory as to whether or
issues with model calibration or other areas. performance and is accurate, not this requires
Use of Thresholds: The absolute values but it is yielding poor results further analysis/
Be Timely: Model monitoring should ideally of most computed metrics are difficult in some of the other metrics. remediation action.
be performed on a monthly basis, especially to interpret without the use of visual signals.
for models involved in real-time decision Traffic Lights for computed metrics are Include Clear and Relevant Insights and
making. While this may seem obvious, cases now increasingly common in most vendor Commentary: Quantitative outcomes are not
have been observed where such models are offerings and can be easily interpreted. self-explanatory; they require interpretation
monitored only in an ad hoc manner; this is These should be defined in the model and explanation, and skills and time are not
especially true of models in areas not subject monitoring standards. always sufficient to do a thorough job. This
to a high degree of regulatory scrutiny. is probably one of the most time-consuming
tasks for the monitoring unit, but it can also
be the most valuable for the organization
if the regular analysis process provides
Figure 4: Elements of a Robust Model Monitoring MIS key findings to senior management.
Overall
Visual Timely
Status
12
13
CONCLUSION
Model monitoring is an area The Federal Reserve, for example, has issued effectiveness of their model monitoring.
guidelines on model monitoring in its SR 11-7 These steps include diligently tracking model
of increasing importance guidance note as well as annual guides on the performance, escalating and resolving
and regulatory scrutiny as CCAR process;4 similar documents have been model issues, involving senior management
issued by the Basel Committee and the Bank in decision making, fine-tuning models
models are treated as critical of England.5 on a timely basis, and maintaining well-
organizational assets. documented logs and rationales of changes.
A strong credit risk model monitoring Effective monitoring allows institutions
process is not only required by regulations, to closely control and better empower the
but it has proven to be a potent competitive strategic risk management tools for day-to-
advantage for those organizations that day operational management as well as for
take extra steps to help ensure the purposes of calculating capital requirements.
14
HOW ACCENTURE CAN HELP
Accenture has extensive In addition to skills related to model and automated dashboards. Figure 5 below
development, validation and monitoring illustrates one such offering, Accentures
experience in all aspects and an understanding of the regulatory Credit Risk Model Monitoring.
of model validation and implications of Basel II and III, Dodd-Frank,
Federal Reserve and UK Prudential Regulation For more information on the topics covered
monitoring, including a team Authority (PRA) requirements (among others), in this document, as well as any general
that has delivered more than Accenture has developed proprietary assets queries on model risk monitoring, please
that can help accelerate and simplify the contact the authors.
500 risk analytics solutions. setting up of a robust model monitoring
process with pre-written modules in SAS
PD Models
Automated
Linkage Underwriting Models
EAD Models
LGD Models