Category: Monitoring & Drift

Ongoing Performance Monitoring

Also known as: Ongoing Monitoring, Continuous Performance Monitoring, Performance Monitoring

Simply put

Ongoing performance monitoring is the continuous process of checking that a model keeps working as intended after it is put into use, rather than only testing it once before deployment. It involves regularly collecting and reviewing metrics over time to catch problems such as declining accuracy or changing conditions. In regulated settings like banking, this monitoring also helps confirm that a model still meets applicable expectations and supervisory requirements.

Formal definition

Ongoing performance monitoring refers to the recurring collection, measurement, and analysis of model-relevant metrics after deployment to confirm that a model continues to perform as intended and to detect degradation, drift, or changes in operating conditions over time. In model risk management, it is commonly treated as a core, continuing component of the model lifecycle that complements—but does not replace—initial validation; monitoring verifies sustained fitness for use, whereas validation assesses conceptual soundness and outcomes at defined points. It is frequently associated with tracking key performance indicators and, in regulated contexts, with verifying continued regulatory compliance and alignment with supervisory or organizational expectations. Note that this activity is generally aimed at reducing and managing model risk rather than eliminating it, and the specific metrics, thresholds, cadence, and governance obligations vary by sector, regulatory regime, and model type; the evidence provided here does not specify particular metrics, thresholds, or the text of any regulatory requirement.

Why it matters

A model that performs well in initial validation can still deteriorate once it is exposed to live conditions. Data distributions shift, populations change, and the relationships a model was built to capture can weaken over time. Ongoing performance monitoring exists to catch these problems as they emerge, rather than discovering them only after a model has been producing unreliable outputs for an extended period. Because monitoring is a continuing activity, it complements initial validation but does not substitute for it: validation assesses conceptual soundness and outcomes at defined points, while monitoring confirms sustained fitness for use over time.

In regulated environments, ongoing monitoring also carries a compliance dimension. In the banking context, ongoing monitoring helps institutions verify continued regulatory compliance and keep models performing as intended within their model risk management processes. U.S. banking supervisory guidance on model risk management (issued jointly by the Federal Reserve and OCC, commonly cited as SR 11-7 / OCC 2011-12) treats ongoing monitoring as an expected component of sound model risk management practice. Separately, the EU Artificial Intelligence Act (Regulation (EU) 2024/1689) establishes post-market monitoring obligations for high-risk AI systems; readers should note this is an EU legal instrument scoped to systems within its coverage and is distinct in nature and jurisdiction from U.S. banking supervisory guidance.

It is important to frame monitoring accurately: it is aimed at reducing and managing model risk, not eliminating it. Detecting degradation early narrows the window in which a failing model can cause harm, but the presence of a monitoring program does not by itself guarantee that all problems will be caught, nor does it remove the need for governance, escalation, and remediation when issues are found. The evidence available here does not specify particular metrics, thresholds, cadences, or the precise text of any regulatory requirement, and these vary by sector, regime, and model type.

Who it's relevant to

Model risk managers and validators

Those responsible for model risk management rely on ongoing monitoring as a continuing lifecycle activity that complements initial validation. It provides the evidence needed to confirm sustained fitness for use, flag degradation, and trigger revalidation or remediation. Practitioners should keep the distinction between monitoring and validation clear rather than treating one as a replacement for the other.

Compliance officers in regulated institutions

In banking and similar regulated settings, ongoing monitoring supports verification of continued regulatory compliance and helps demonstrate that models keep performing as intended. U.S. banking supervisory guidance (SR 11-7 / OCC 2011-12) treats it as an expected element of model risk management, so compliance teams often need to evidence that monitoring is performed and acted upon.

Teams deploying high-risk AI systems under the EU AI Act

Providers and deployers of high-risk AI systems within the scope of the EU AI Act (Regulation (EU) 2024/1689) face post-market monitoring obligations. For these teams, ongoing monitoring is not only a risk-management practice but connects to legal requirements specific to that regime and jurisdiction, which differ in nature from U.S. supervisory guidance.

Data scientists and model developers

Developers are typically expected to establish a monitoring plan and begin regular monitoring once a model enters use. They help define which metrics are tracked and how observed behavior is compared against expectations, giving downstream governance functions the signals needed to detect drift or declining accuracy.

Internal auditors and oversight functions

Auditors and second- and third-line functions assess whether monitoring is designed appropriately, performed consistently, and escalated when thresholds are breached. They focus on whether the program actually reduces and manages model risk in practice, recognizing that monitoring mitigates rather than eliminates risk.

Inside Ongoing Performance Monitoring

Performance metric tracking

The systematic measurement of model outputs against expected behavior over time, typically covering accuracy, discrimination, calibration, or other outcome measures relevant to the model's use case. This component distinguishes ongoing monitoring from one-time validation by emphasizing continuous observation across the model's operational life.

Threshold and trigger definition

Predefined tolerance bands or breach conditions that, when exceeded, prompt review, escalation, or remediation. As commonly framed, these thresholds link observed performance to defined governance actions so that degradation does not go unaddressed.

Input and population stability assessment

Monitoring of shifts in the input data distribution or the population to which the model is applied (often discussed as data drift or population stability). Such shifts can signal that assumptions embedded at development time no longer hold, even when headline performance appears stable.

Model performance degradation detection

The identification of decline in a model's predictive or operational quality over time. Note that model performance degradation is a distinct concept from model risk more broadly: degradation is one driver of risk, but model risk also encompasses fundamental errors, misuse, and inappropriate application that monitoring metrics may not capture.

Escalation and reporting pathways

The routing of monitoring findings to accountable parties, connecting the activity to organizational oversight structures. This is where ongoing monitoring intersects AI governance, though the monitoring activity itself is more precisely a model risk management control.

Documentation and audit trail

Records of monitoring results, methods, thresholds, and any actions taken. Such records support review by second-line functions (independent risk oversight) and third-line functions (internal audit), and support supervisory or regulatory examination where applicable.

Common questions

Answers to the questions practitioners most commonly ask about Ongoing Performance Monitoring.

Is ongoing performance monitoring the same thing as model validation?

No, though they are closely related and often confused. Validation is a broader set of activities intended to establish that a model is conceptually sound and fit for its intended use, and in many model risk management frameworks ongoing monitoring is treated as one component of the validation lifecycle rather than a separate discipline. Monitoring specifically refers to the continued observation of a model's behavior after deployment. Under U.S. banking supervisory guidance (SR 11-7 / OCC 2011-12), ongoing monitoring is described as an integral part of model validation, so the two are linked rather than interchangeable. Treating monitoring as a substitute for periodic revalidation, or vice versa, is a common error.

Does ongoing performance monitoring only track predictive accuracy?

Not typically. While accuracy or predictive performance is a frequent focus, monitoring as commonly defined can also cover input data quality and distribution shifts, stability of outputs, changes in the operating environment, and adherence to assumptions and limitations documented at development. It is worth distinguishing model risk from model performance degradation: a decline in a performance metric is one signal, but monitoring is often designed to surface a wider range of risks. The precise scope varies by framework and by the risk profile of the model, so reducing monitoring to a single accuracy dashboard usually understates its intended role.

How often should ongoing performance monitoring be performed?

There is no single universally required frequency, and it generally depends on the model's risk rating, the volatility of its inputs, and applicable internal policy or regulatory expectations. In many frameworks, higher-risk or high-materiality models are monitored more frequently, while lower-risk models may be reviewed on a longer cycle. Some indicators may be tracked continuously or in near real time, while others are assessed at defined intervals. Firms typically document the rationale for chosen frequencies rather than applying a uniform schedule across all models.

What metrics or thresholds should trigger escalation or remediation?

This is generally set through predefined monitoring metrics and thresholds tied to the model's intended use and risk tolerance. Common practice is to define acceptable ranges in advance and to establish escalation paths when thresholds are breached, but the specific metrics vary by model type and objective. Thresholds are often calibrated to distinguish routine variation from meaningful deterioration, and breaches typically prompt investigation before remediation, since a threshold breach may reflect a data issue rather than a genuine model failure. The appropriate metrics and limits should be documented and periodically reviewed rather than fixed permanently.

Who is responsible for ongoing performance monitoring within an organization?

Responsibilities are commonly distributed across the lines of defense, though the exact allocation depends on the organization's governance structure. In many arrangements, model owners or developers in the first line perform or feed routine monitoring, an independent second-line function reviews or challenges the monitoring, and internal audit in the third line provides periodic assurance over the process. It is important not to collapse these roles: the party running a model and the party independently challenging its performance are typically kept distinct to preserve effective oversight. Clear ownership and documented handoffs are generally expected.

How should ongoing performance monitoring be documented and reported?

Documentation typically captures what is being monitored, the metrics and thresholds used, the frequency, the results over time, and any actions taken in response to findings. Reporting is often structured so that monitoring outcomes reach appropriate governance or oversight bodies, with escalation for significant issues. Maintaining an auditable record supports both internal review and external examination, and the level of detail is commonly scaled to the model's risk and materiality. Firms generally treat monitoring documentation as an evolving record rather than a one-time artifact.

Common misconceptions

Ongoing performance monitoring is essentially the same as validation, so if a model was validated it does not need continuous monitoring.

Validation and ongoing monitoring are distinct activities that experts do not blur. Validation is a point-in-time (or periodic) evaluation of model soundness, while ongoing monitoring is the continuous tracking of behavior during operation. In many frameworks—including U.S. banking supervisory guidance SR 11-7 / OCC 2011-12—ongoing monitoring is treated as an integral component of model validation and lifecycle risk management rather than a substitute for it.

Ongoing performance monitoring is a voluntary best practice with no regulatory basis.

Monitoring during operation is required or expected under specific frameworks within their respective jurisdictions. U.S. banking supervisory guidance SR 11-7 / OCC 2011-12 identifies ongoing monitoring as a core element of model risk management for supervised institutions, and the EU Artificial Intelligence Act (Regulation (EU) 2024/1689) establishes post-market monitoring obligations for high-risk AI systems. These instruments differ in legal character and scope—supervisory guidance versus binding regulation—and are not interchangeable or universally applicable, so practitioners should scope obligations to the relevant regime.

Stable headline performance metrics mean the model is safe and no action is needed.

Aggregate performance can remain stable while input distributions shift, subpopulation outcomes degrade, or the model is applied outside its intended use. Monitoring reduces and helps manage model risk but does not eliminate it, and it should examine drift and segment-level behavior rather than relying on a single top-line metric.

Best practices

Define monitoring metrics, tolerance thresholds, and escalation triggers before deployment, and document how each triggered outcome maps to a specific governance action.

Monitor input and population stability (drift) alongside outcome metrics, since aggregate performance can mask localized or subpopulation degradation.

Scope the monitoring program to the applicable framework—for example SR 11-7 / OCC 2011-12 expectations for supervised banking institutions or EU AI Act post-market monitoring for high-risk systems—rather than assuming a single universal standard applies.

Maintain an auditable trail of monitoring results, methods, and remediation actions so that independent second-line oversight and third-line internal audit can review the activity.

Route findings through defined escalation and reporting pathways to accountable parties, keeping the distinction clear between monitoring as a model risk management control and the broader governance structures that receive its output.

Treat monitoring as a measure that reduces and manages model risk rather than one that eliminates it, and pair it with periodic revalidation to catch issues that continuous metrics may not surface.