classifying something as a model carries with it important consequences related to cost and productivity.
procedures can be both expensive for the institution and onerous for the model owner.
Model validations pull model owners away from their everyday work, adversely affecting productivity and, sometimes, quality of work.
Too many unnecessary reports containing findings that are comparatively unimportant can bury enterprise risk managers and distract them from the most urgent findings.
Definition of a model
A quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates
Models as have three components
An information input component
A processing component,
A reporting component,
Quantitative estimate implies a level of uncertainty about the outputs.
On Spreadsheet models
Probably two types:
a model that transforms inputs into quantitative estimates or
a non-model spreadsheet that generates defined arithmetic results.
Ask oneself if back-testing required to gauge the accuracy of the spreadsheet’s outputs
Ask if the spreadsheet is simply applying a defined set of business rules (then not a model)
Spreadsheets should be classified as models (and validated as such) when they produce forward-looking estimates that can be back-tested.
On vendor models
Vendor documentation is not a substitute for model documentation:
Documentation should cover:
Discussion of the model’s purpose and specific application,
Discussion of model theory and approach,
Description of the model’s structure
Identification of model limitations and weaknesses
Comprehensive list of inputs and assumptions, including their sources
Comprehensive list of outputs and reports
List Model settings
Description of testing
Testing results should be requested of the vendor: (per OCC 2011-12) including performance monitoring and outcomes analysis,
Model validators should attempt to replicate the results of these studies, where feasible
Developmental evidence should be requested of the vendor: (per OCC 2011-12)
Banks should have contingency plans for the vendor model
Implementation of reliable challenger model (for contingency)
On general model documentation
Model validations frequently seem to occur at the most inopportune moments for model owners.
Documentation would ideally be prepared during periods of lower operational stress.
Documentation follows these basic criteria:
Identifies the model’s purpose,
Comprehensively lists and justifies the model’s inputs and assumptions
Describes the model’s overall theory and approach,
Lays out the developmental evidence supporting choices
Identifies the limitations of the model
Explains how the model is controlled—who
Comprehensively identifies and describes the model’s outputs, how they are used, and how they are tested
On Model validation
Comprehensive model validations consist of three main components:
Conceptual soundness
Monitoring and benchmarking,
Outcomes analysis and back-testing.
Performance testing is the core of any model validation
Benchmarking
Back-testing
Sensitivity analysis
Stress testing
Validate model inputs based on:
Complexity of inputs
Manual manipulation of inputs from source system prior to input into model
Reliability of source system
Relative importance of the input to the model’s outputs (i.e., sensitivity)
On Benchmarking and Back-testing
Model developers as best practice should always list any alternative methodologies, theories, or data which were omitted from the model’s final version.
Back-testing measures a model’s outcome and accuracy against real-world observations,
Benchmarking measures those outcomes against those of other models or metrics.
Back-testing and Benchmarking should ideally be performed together
