A validator’s guide to model risk management

thumbnail of A Validator’s Guide to Model Risk Management_RiskSpan Ebook Original by Riskspan, 2017, 28 pages 

This summary note was Posted on

Things to consider

  • classifying something as a model carries with it important consequences related to cost and productivity.
  • procedures can be both expensive for the institution and onerous for the model owner.
  • Model validations pull model owners away from their everyday work, adversely affecting productivity and, sometimes, quality of work.
  • Too many unnecessary reports containing findings that are comparatively unimportant can bury enterprise risk managers and distract them from the most urgent findings.

Definition of a model

A quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates

  • Models as have three components
    • An information input component
    • A processing component,
    • A reporting component,
  • Quantitative estimate implies a level of uncertainty about the outputs.

On Spreadsheet models

  • Probably two types:
    • a model that transforms inputs into quantitative estimates or
    • a non-model spreadsheet that generates defined arithmetic results.
  • Ask oneself if back-testing required to gauge the accuracy of the spreadsheet’s outputs
  • Ask if the spreadsheet is simply applying a defined set of business rules (then not a model)
  • Spreadsheets should be classified as models (and validated as such) when they produce forward-looking estimates that can be back-tested.

On vendor models

  • Vendor documentation is not a substitute for model documentation:
  • Documentation should cover:
    • Discussion of the model’s purpose and specific application,
    • Discussion of model theory and approach,
    • Description of the model’s structure
    • Identification of model limitations and weaknesses
    • Comprehensive list of inputs and assumptions, including their sources
    • Comprehensive list of outputs and reports
    • List Model settings
    • Description of testing
  • Testing results should be requested of the vendor: (per OCC 2011-12) including performance monitoring and outcomes analysis,
  • Model validators should attempt to replicate the results of these studies, where feasible
  • Developmental evidence should be requested of the vendor: (per OCC 2011-12)
  • Banks should have contingency plans for the vendor model
  • Implementation of reliable challenger model (for contingency)

On general model documentation

  • Model validations frequently seem to occur at the most inopportune moments for model owners.
  • Documentation would ideally be prepared during periods of lower operational stress.
  • Documentation follows these basic criteria:
    • Identifies the model’s purpose,
    • Comprehensively lists and justifies the model’s inputs and assumptions
    • Describes the model’s overall theory and approach,
    • Lays out the developmental evidence supporting choices
    • Identifies the limitations of the model
    • Explains how the model is controlled—who
    • Comprehensively identifies and describes the model’s outputs, how they are used, and how they are tested

On Model validation

  • Comprehensive model validations consist of three main components:
    • Conceptual soundness
    • Monitoring and benchmarking,
    • Outcomes analysis and back-testing.
  • Performance testing is the core of any model validation
    • Benchmarking
    • Back-testing
    • Sensitivity analysis
    • Stress testing
  • Validate model inputs based on:
    • Complexity of inputs
    • Manual manipulation of inputs from source system prior to input into model
    • Reliability of source system
    • Relative importance of the input to the model’s outputs (i.e., sensitivity)

On Benchmarking and Back-testing

  • Model developers as best practice should always list any alternative methodologies, theories, or data which were omitted from the model’s final version.
  • Back-testing measures a model’s outcome and accuracy against real-world observations,
  • Benchmarking measures those outcomes against those of other models or metrics.
  • Back-testing and Benchmarking should ideally be performed together