Skip to main content
Bureau of AI
Category: Validation & Testing

Model Calibration

Also known as: Calibration
Simply put

Model calibration is the process of adjusting a model so its outputs line up with what actually happens in the real world. For example, when a model that predicts probabilities says an event has a 70% chance, a well-calibrated model means that such events truly occur about 70% of the time. The specific meaning varies by context, ranging from tuning simulation parameters to match observed data to correcting predicted probabilities.

Formal definition

Model calibration refers to adjusting a model so that its outputs are consistent with observed real-world outcomes, though the precise objective depends on the modeling context. In probabilistic and machine learning settings, calibration typically means adjusting predicted probabilities so that estimated class probabilities reflect the true likelihood of the event—i.e., predicted probabilities align with observed empirical frequencies. In simulation, forecasting, and parametric modeling contexts, calibration typically refers to adjusting parameter values until predicted quantities match observed data within a defined domain (for example, aligning predicted travel with observed travel, occupancy predictions with observed occupancy, or media mix predictions with incrementality test results). Note that the evidence provided describes calibration primarily in general modeling and machine learning terms; it does not establish domain-specific regulatory usage, and calibration should be distinguished from related concepts such as model validation and model performance more broadly, which are out of scope here.

Why it matters

Model calibration matters because a model can appear accurate in aggregate while still producing outputs that misstate the likelihood of specific events. A classifier that separates outcomes well may still assign probabilities that do not reflect true empirical frequencies—so when it reports a 70% chance, such events may occur substantially more or less often. In decision contexts where the probability itself drives action, such as thresholds, resource allocation, or downstream calculations, poorly calibrated outputs can lead to systematically flawed decisions even when the model's discrimination or ranking appears strong.

Because the objective of calibration varies by modeling context, the stakes also vary. In probabilistic and machine learning settings, calibration concerns whether predicted probabilities align with observed outcomes. In simulation, forecasting, and parametric contexts, calibration concerns whether tuned parameters produce predictions that match observed data—such as aligning predicted travel with observed travel, predicted occupancy with observed occupancy, or media mix predictions with incrementality test results. Treating these as a single uniform practice can obscure what a given calibration effort is actually verifying.

It is important to distinguish calibration from broader model validation and from model performance more generally, which are out of scope here. A well-calibrated model is not automatically a valid or fit-for-purpose model, and calibration reduces one specific type of error rather than eliminating model risk. The evidence base describes calibration in general modeling and machine learning terms and does not establish domain-specific regulatory usage; readers in regulated settings should confirm how calibration is defined and expected within their own framework.

Who it's relevant to

Model Risk Managers and Validators
Calibration is one dimension a validator may assess when examining whether a model's outputs correspond to observed outcomes. It should be treated as distinct from broader model validation and overall model performance, and confirming calibration does not by itself establish that a model is valid or fit for purpose.
Data Scientists and Modelers
Practitioners building probabilistic or machine learning models use calibration to adjust predicted probabilities so estimated class probabilities reflect the true likelihood of an event. Those working with simulation, forecasting, or parametric models use it to tune parameter values until predictions match observed data within the relevant domain.
Forecasting and Simulation Practitioners
In simulation and parametric modeling, calibration is the process of adjusting parameters until predicted quantities—such as predicted travel or occupancy—match observed measurements within a defined study area. Here calibration is about parameter tuning against observed data rather than probability correction.
Marketing and Measurement Analysts
In media mix modeling contexts, calibration can refer to adjusting model predictions to be consistent with conclusions drawn from incrementality tests, using experimental evidence as the real-world reference point against which the model is aligned.

Inside Model Calibration

Predicted probability alignment
Model calibration concerns the degree to which a model's predicted probabilities (or scores) correspond to observed empirical frequencies. A well-calibrated model that assigns a 0.7 probability to a set of cases should, in aggregate, see the modeled event occur in approximately 70% of those cases. This is a property of the probability estimates, distinct from whether the model correctly ranks or classifies outcomes.
Calibration assessment methods
Commonly used tools include reliability (calibration) plots that compare predicted probabilities against observed frequencies across bins, and summary statistics or tests intended to quantify miscalibration. The choice of method and binning approach can affect conclusions, so results are typically interpreted alongside the modeling context rather than in isolation.
Recalibration techniques
When miscalibration is identified, post-hoc adjustment methods may be applied to the model's outputs to better align predicted probabilities with observed frequencies. These techniques adjust the mapping from scores to probabilities and are generally distinct from retraining the underlying model.
Relationship to discrimination and performance
Calibration is one dimension of model quality and is conceptually separate from discrimination (the ability to distinguish between outcome classes, often summarized by rank-order measures). A model can rank cases well yet produce poorly calibrated probabilities, and vice versa, so both dimensions are typically examined.
Placement within model risk practice
In many model risk management contexts, calibration checks form part of ongoing monitoring and validation activities, where deteriorating calibration over time may signal that a model's probability estimates no longer reflect current conditions.

Common questions

Answers to the questions practitioners most commonly ask about Model Calibration.

Is model calibration the same thing as model accuracy?
No. Calibration and accuracy are distinct properties, and professionals frequently conflate them. Calibration refers to how well a model's predicted probabilities align with observed outcome frequencies—for example, whether events assigned a 70% probability actually occur roughly 70% of the time over many such predictions. Accuracy typically measures how often the model's discrete classifications or point predictions are correct. A model can be well-calibrated yet have poor discriminatory accuracy, or highly accurate at classification while producing poorly calibrated probability estimates. Because these measure different things, they should be assessed separately rather than treated as interchangeable.
Does calibrating a model correct for underlying bias or fairness problems?
Not necessarily, and treating calibration as a fairness remedy is a common error. Calibration concerns the reliability of predicted probabilities relative to observed outcomes; it does not by itself address whether the training data, features, or outcomes embed systematic disparities. A model can be well-calibrated in aggregate while still producing disparate results across subgroups, and calibration adjustments made for one population may not hold for another. Calibration and fairness assessment are separate analyses, and calibration should not be presented as a substitute for bias testing or fairness evaluation.
How is model calibration typically assessed in practice?
Calibration is commonly assessed by comparing predicted probabilities against observed outcome frequencies. Practitioners often group predictions into bins and compare the average predicted probability in each bin with the actual observed frequency, and may visualize this relationship using reliability diagrams. Various summary metrics are also used to quantify the gap between predicted and observed frequencies. The specific approach and thresholds depend on the model type, use case, and any applicable internal or regulatory expectations, so no single method is universally required.
When should calibration be re-assessed after a model is deployed?
Calibration is generally treated as something to monitor on an ongoing basis rather than verify only at initial validation, because the relationship between predicted probabilities and observed outcomes can shift over time. Re-assessment is commonly triggered by changes in the input population, shifts in the operating environment, or observed model performance degradation, as well as on a periodic schedule defined by an organization's monitoring policy. The appropriate cadence depends on the model's risk profile and how frequently outcome data becomes available.
Where does calibration fit within a broader model validation and monitoring process?
Calibration assessment is typically one component within model validation and ongoing monitoring rather than a standalone activity. It commonly sits alongside evaluations of discriminatory performance, stability, and conceptual soundness. In many organizational structures, initial calibration testing may occur during validation activities associated with a second line of defense, while ongoing calibration monitoring may involve model owners and other functions. The precise allocation of responsibilities depends on the institution's governance structure and applicable expectations.
What should be documented when reporting on a model's calibration?
Documentation practices vary by organization and use case, but reporting on calibration commonly includes the method used to assess it, the data and time period the assessment covers, the results relative to any predefined tolerances or thresholds, and any limitations of the analysis. Where calibration was adjusted, the adjustment approach and the population it was based on are typically recorded so that reviewers can understand whether it may generalize to other populations. Because appropriate documentation depends on internal policy and any applicable regulatory or standards-based expectations, the specific requirements should be confirmed against the relevant framework.

Common misconceptions

A model with high predictive accuracy or strong rank-ordering is necessarily well calibrated.
Calibration and discrimination are distinct properties. A model can separate classes effectively while still producing predicted probabilities that systematically over- or under-state observed frequencies. As commonly framed, both should be assessed rather than inferring one from the other.
Recalibration fixes an underlying model problem.
Post-hoc recalibration adjusts the mapping from scores to reported probabilities; it does not necessarily address deficiencies in the model's structure, inputs, or ability to discriminate. Miscalibration can also be a symptom of population or environment change, which recalibration alone may not resolve.
Calibration is a one-time check performed at development.
Because the relationship between predicted probabilities and observed frequencies can drift as underlying conditions change, calibration is typically treated as something to be monitored on an ongoing basis rather than established once and assumed stable.

Best practices

Assess calibration separately from discrimination, and report both so that stakeholders understand that good ranking does not guarantee accurate probabilities.
Use reliability plots alongside summary measures, and document the binning or grouping choices made, since these decisions can influence the apparent degree of miscalibration.
Evaluate calibration on data that reflects the intended use population and time period, and be cautious about generalizing calibration results to segments or conditions not represented in the assessment sample.
Incorporate calibration checks into ongoing monitoring so that drift in the alignment between predicted probabilities and observed frequencies can be detected over time.
When applying recalibration, treat it as an adjustment to reported probabilities and investigate whether observed miscalibration reflects a deeper model or data issue rather than assuming the adjustment resolves the root cause.
Document the calibration methodology, thresholds for action, and any recalibration steps as part of model documentation and validation records, consistent with the organization's model risk practices.