Skip to main content
Bureau of AI
Category: Model Lifecycle & MLOps

Model Serving

Also known as: Model Deployment (in the serving sense), Inference Serving, ML Serving
Simply put

Model serving is the practice of making a trained machine learning model available for use by hosting it in a production environment, typically behind an API or endpoint, so that applications can send in new data and receive predictions or outputs. It is the step that turns a model from something built and tested into something that can actually be used in day-to-day operations. Serving can take place in the cloud or on an organization's own infrastructure.

Formal definition

Model serving is the runtime layer that deploys a trained ML model as a network-invokable service, commonly exposed through stable APIs or inference endpoints, enabling applications to submit input data and obtain inference results. It encompasses hosting the model (on cloud or on-premises infrastructure) and, as commonly described in MLOps practice, managing operational concerns such as latency, scaling, and versioning. Note that in some sources the term overlaps with 'model deployment,' but serving specifically refers to the ongoing availability of model functions for inference rather than the full deployment lifecycle; from a governance and model risk perspective, serving is where a model's outputs affect live decisions and therefore where monitoring and control measures typically apply, though the evidence here does not detail those controls.

Why it matters

Model serving is the point at which a machine learning model stops being an internal artifact and begins influencing live decisions. As the evidence describes, serving is the runtime layer that connects models to live data, applications, and decisions. From a governance and model risk perspective, this is significant: it is where a model's outputs actually reach downstream applications and, potentially, the people or processes those applications affect. A model that performs well in testing but is served without adequate oversight can still produce harmful or erroneous outputs in production.

Because serving is where inference happens against new, real-world inputs, it is typically the locus where operational concerns such as latency, scaling, and versioning must be managed, as noted in the evidence. It is also, as commonly framed in MLOps practice, the natural place to situate monitoring and control measures, since this is where the gap between a model's tested behavior and its live behavior becomes observable. The evidence digest here does not detail those specific controls, so this entry describes serving as the layer at which such measures typically apply rather than prescribing a particular control framework.

Who it's relevant to

Data Scientists and ML Engineers
These practitioners are typically responsible for preparing trained models for serving and configuring the endpoints that expose them. Serving is where their models transition into ongoing production use, making decisions about hosting, versioning, and scaling directly relevant to their work.
Model Risk Managers
Because serving is where a model's outputs reach live decisions, it is a focal point for identifying and managing model risk. Model risk managers may treat the serving layer as the place where monitoring and control measures typically apply, though the specific design of those measures falls outside the scope of this entry.
Compliance and Governance Specialists
Serving is the stage at which model outputs begin to affect operational decisions, which is often where governance obligations and accountability structures attach. Understanding where serving sits in the lifecycle helps these professionals scope oversight to the point of live use rather than to the model artifact alone.
IT and Infrastructure Teams
Serving can take place in the cloud or on an organization's own infrastructure, and it involves managing operational concerns such as latency and scaling. Infrastructure teams are therefore relevant to how reliably and efficiently models remain available for inference in production.

Inside Model Serving

Serving Infrastructure
The runtime environment and compute resources that host a trained model and expose it for inference, whether through real-time (synchronous) endpoints, batch processing, or streaming pipelines. The choice of serving pattern typically depends on latency requirements and use case.
Inference Endpoint / API
The interface through which consuming applications or users submit inputs and receive model outputs. This is commonly where access controls, authentication, and request logging are applied.
Input and Output Handling
The preprocessing of incoming data and postprocessing of model outputs at serving time. Discrepancies between the data transformations used in training and those applied at serving (often called training-serving skew) are a recognized source of degraded model performance in production.
Version and Deployment Management
Controls governing which model version is active, how new versions are promoted, and how rollbacks occur. Deployment strategies such as staged rollouts or parallel comparison are commonly used to manage the risk of introducing a new version.
Monitoring and Logging
The capture of operational metrics (latency, throughput, availability) and model-behavior signals (input distributions, output distributions, prediction quality where ground truth is available). Monitoring at the serving layer is a key input to detecting model performance degradation over time, which is distinct from model risk itself.
Governance and Accountability Linkage
The connection between the serving layer and broader AI governance structures, such as documented ownership, approval before deployment, and defined responsibilities across lines of defense. Serving is where governance policies and model risk controls are operationally enforced, though the two remain conceptually distinct.

Common questions

Answers to the questions practitioners most commonly ask about Model Serving.

Is model serving the same as model deployment?
Not exactly, though the terms are often used loosely and overlap in practice. Model serving typically refers to the runtime function of exposing a trained model so that it can respond to inference requests, whereas deployment is commonly used more broadly to cover the process of releasing a model into an operational environment, which may include packaging, promotion through environments, configuration, and serving as one component. Treating them as strictly synonymous can obscure where governance controls and accountability attach. The distinction is not standardized across all organizations, so definitions should be confirmed against your own operational documentation.
Does having a model in serving mean it has been validated and its risks controlled?
No. The fact that a model is served and responding to requests speaks to its operational availability, not to whether independent validation has occurred or whether its risks have been assessed and controlled. In many model risk frameworks, validation and the establishment of monitoring and controls are distinct activities from making a model available for inference. Serving infrastructure can technically expose a model regardless of its governance status, so serving should not be read as evidence that oversight requirements have been met.
What monitoring is typically associated with a model that is in serving?
In many practices, models in serving are monitored for operational metrics such as availability, latency, and error rates, and separately for indicators relevant to model risk, which may include input distribution shifts, output patterns, and signs of performance degradation over time. The specific monitoring depends on the model's use, its inherent risk, and applicable internal policy. It is worth distinguishing operational monitoring of the serving system from monitoring that supports model risk management, as the two answer different questions and are sometimes owned by different teams.
How does model serving relate to the lines-of-defense structure?
Responsibilities around serving are often distributed across the lines of defense, though the precise allocation varies by organization. The teams that build, operate, and serve a model are commonly associated with the first line, while independent review or validation functions and risk oversight are commonly associated with the second line, and internal audit with the third. The serving activity itself is an operational function; the controls, monitoring, and independent challenge around it are what map onto the lines-of-defense model. You should confirm how your organization assigns these roles rather than assume a universal arrangement.
What considerations arise when serving multiple versions of a model at once?
Serving more than one version, for example during a phased rollout or comparison, typically raises questions about which version is authoritative for a given decision, how outputs are attributed to a specific version for traceability, and how each version's governance status is tracked. Clear version identification and logging are commonly emphasized so that decisions can be linked to the model that produced them. The appropriate approach depends on the use case and internal policy, and this entry does not prescribe a specific configuration.
What should be logged when a model is in serving?
Logging practices vary, but organizations commonly capture information supporting traceability and auditability, which may include the model version invoked, timestamps, and relevant inputs and outputs, subject to applicable privacy and data-handling constraints. Such records can support monitoring, investigation of issues, and review by oversight functions. What is actually required depends on the model's risk profile, applicable internal policy, and any governing obligations, so logging design should be confirmed against those sources rather than treated as fixed.

Common misconceptions

Once a model is validated and deployed to serving, the model risk work is largely complete.
Deployment is a point in an ongoing lifecycle. Model behavior can change relative to expectations as real-world data shifts, so ongoing monitoring of the served model is typically treated as a continuing control. Note also that validation (assessing whether the model is conceptually sound and fit for purpose) is distinct from verification (confirming the deployed system behaves as specified), and serving relates to both.
Model serving and model performance degradation are the same concern.
Serving is the operational function of exposing a model for inference; performance degradation is a phenomenon that a served model may exhibit over time. Serving infrastructure is often where degradation is observed and measured, but the serving layer functioning correctly does not guarantee the model's predictive quality remains adequate.
Governance controls implemented at the serving layer eliminate model risk.
Serving-layer controls such as access management, version control, and monitoring reduce and help manage risk; they do not remove it. Residual risk typically remains even after controls are applied, and its acceptability is a governance judgment rather than a property established solely by the serving setup.

Best practices

Align preprocessing and postprocessing logic between training and serving environments to reduce training-serving skew, and test for consistency before promoting a new version.
Instrument serving endpoints to capture both operational metrics (latency, availability, throughput) and model-behavior signals (input and output distributions) so that performance degradation can be detected as early as available data allows.
Maintain explicit version and deployment management, including documented approval before a model version becomes active and a tested rollback path.
Use staged deployment strategies to limit exposure when introducing new versions, and define measurable criteria for promoting or reverting a version.
Enforce access controls, authentication, and request logging at the inference endpoint, and link these controls to the responsibilities defined across the relevant lines of defense.
Treat post-deployment monitoring as an ongoing control tied to documented ownership rather than a one-time check, and escalate deviations through established governance channels.