Category: Validation & Testing

A/B Testing

Also known as: bucket testing, split-run testing, split testing

Simply put

A/B testing is a method for comparing two versions of something—such as a webpage, app feature, or marketing message—to see which one performs better with users. Users are typically split into groups, with each group shown a different version, and the results are measured to inform a decision. It is commonly used to produce data-driven insights rather than relying on assumptions.

Formal definition

A/B testing is a controlled comparison method in which an audience is divided into groups, each exposed to a different version of a variable (for example, versions A and B of a webpage or app), so that performance against a defined outcome can be measured and compared. As described in the evidence, it is applied in user-experience research and marketing contexts to compare two versions of a product or campaign element and determine which performs better. The evidence provided does not detail the underlying statistical methodology (such as significance testing, sample sizing, or randomization procedures), so this entry is scoped to the general purpose and structure of the technique as commonly presented; note that A/B testing as used for model or algorithm evaluation may involve additional considerations not covered here.

Why it matters

A/B testing matters because it replaces assumptions with evidence. As described in the evidence, it produces hard data that helps teams make informed, effective decisions rather than relying on intuition about what users prefer. For organizations building AI-enabled products or optimizing user-facing experiences, this shift toward data-driven decision-making is a foundational discipline.

In AI governance and model risk contexts, A/B testing is often invoked as a way to compare competing versions of a system in production—for example, two versions of a webpage, feature, or algorithm-driven experience. It is important to note, however, that the evidence provided here scopes A/B testing to user-experience research and marketing use, and does not address the additional considerations that arise when the technique is applied to model or algorithm evaluation. Practitioners should not assume that a marketing-style split test satisfies the validation, monitoring, or fairness assessment expectations that may apply to models under model risk management or AI governance frameworks.

Because the evidence does not detail underlying statistical methodology such as significance testing, sample sizing, or randomization, readers should treat A/B testing as a general comparison structure rather than a complete evaluation methodology. Where results feed decisions about consequential systems, the design and interpretation of the test typically warrant additional scrutiny beyond what the term itself implies.

Who it's relevant to

User-Experience Researchers

A/B testing is described in the evidence as a user-experience research method used to compare two versions of a product and determine which is more compelling for users. Researchers use it to ground design decisions in observed user behavior rather than assumptions.

Marketing Teams

According to the evidence, marketers use A/B testing to split their audience and create multiple versions of a variable in a campaign, comparing which performs better. It is presented as a way to produce hard data that supports informed, effective decisions about campaign elements.

Product and Optimization Teams

Teams responsible for webpages, apps, and product features use A/B testing to compare two versions of the same experience and identify the better performer. This supports iterative, evidence-based product decisions.

Data Scientists and Model Evaluators

A/B testing is sometimes extended to compare versions of models or algorithms in production. The evidence here does not cover the additional statistical and governance considerations this can require, so practitioners in this area should treat the general definition as a starting point rather than a complete evaluation approach.

Inside A/B Testing

Control and Treatment Groups

A/B testing typically divides subjects (such as users, requests, or model outputs) into at least two groups: a control group exposed to the existing baseline (for example, a current model or configuration) and one or more treatment groups exposed to a variant. In an AI context, this often means comparing a candidate model or decision policy against an incumbent.

Randomized Assignment

As commonly defined, assignment of subjects to control and treatment groups is randomized to reduce systematic differences between groups, so that observed differences in outcomes can be more credibly attributed to the variant being tested rather than to confounding factors.

Outcome Metrics

Predefined metrics used to compare groups. These may include performance-oriented measures (for example, accuracy, click-through, conversion, or task completion) and, where relevant, operational or risk-relevant measures. The choice of metric determines what the test can and cannot demonstrate.

Statistical Analysis and Significance

Techniques used to determine whether observed differences between groups are likely to reflect a real effect rather than random variation. This typically involves hypothesis testing, effect size estimation, and consideration of sample size and test duration.

Experimental Scope and Duration

The defined population, time window, and traffic allocation over which the test runs. Scope constrains the generalizability of results; conclusions apply to the tested conditions and may not transfer to other populations, time periods, or deployment contexts.

Common questions

Answers to the questions practitioners most commonly ask about A/B Testing.

Is A/B testing a form of model validation?

No, and conflating the two is a common error. A/B testing typically compares the real-world outcomes of two or more variants (for example, two model versions or configurations) under live conditions to observe which performs better against a chosen metric. Model validation, as commonly framed in model risk management guidance such as SR 11-7, is a broader, independent process assessing conceptual soundness, ongoing monitoring, and outcomes analysis. A/B testing may inform the outcomes-analysis component of validation, but it does not substitute for validation as a whole.

Does a statistically significant A/B test result mean the winning variant is genuinely better?

Not necessarily. Statistical significance addresses the likelihood that an observed difference is not due to chance under the test's assumptions; it does not by itself establish practical importance, durability of the effect, or absence of confounding. Results can be sensitive to sample size, test duration, segment composition, and how success metrics were defined. Professionals often err by treating a significant result as conclusive without considering effect size, business relevance, or whether the effect persists over time.

How should success metrics be defined before running an A/B test?

Metrics are typically specified in advance to reduce the risk of post hoc rationalization. This commonly involves selecting a primary metric aligned to the decision at hand, along with guardrail metrics intended to detect unintended harms or degradation elsewhere. Defining metrics, hypotheses, and decision thresholds before the test begins helps limit selective interpretation of results.

How do you determine an appropriate sample size and test duration?

Sample size and duration are generally driven by the smallest effect worth detecting, the desired confidence and power levels, and the baseline variability of the metric. Duration often needs to span representative cycles (for example, covering different days or usage patterns) so that results are not skewed by short-term or non-representative conditions. Ending a test early once a desired result appears can invalidate the statistical assumptions.

What governance controls are relevant when A/B testing AI models in production?

Because A/B testing exposes real users or decisions to model variants, organizations often apply controls such as documented test plans, predefined rollback criteria, monitoring of guardrail metrics, and review of potential harms before deployment. These measures manage and reduce risk rather than eliminate it. The specific controls depend on the sector and the organization's governance framework, and requirements differ across contexts.

How should conflicting or ambiguous A/B test results be handled?

When results are inconclusive, mixed across segments, or fall within the margin of uncertainty, a common practice is to avoid forcing a decision from a single test. Options may include extending the test, re-examining whether the metric captured the intended outcome, investigating segment-level differences, or treating the result as not supporting a change. Documenting the interpretation and decision rationale supports later review and auditability.

Common misconceptions

A/B testing validates a model in the model risk management sense.

A/B testing typically measures relative live performance or business outcomes between variants under specific conditions. This is distinct from model validation as commonly framed in model risk management contexts (such as guidance historically associated with SR 11-7), which involves independent evaluation of conceptual soundness, ongoing monitoring, and outcomes analysis. A/B test results may inform, but generally do not substitute for, validation activities, and they do not by themselves establish governance controls or documentation.

A statistically significant A/B result means the better-performing variant is lower risk.

Statistical significance addresses whether an observed difference is unlikely to be due to chance for the measured outcome metric; it does not by itself speak to model risk, fairness across subgroups, robustness outside the tested conditions, or residual risk after controls. A variant can improve a headline metric while introducing or concealing other risks that the test was not designed to detect.

A/B testing eliminates the risk of deploying a worse variant.

A/B testing is a measure that can reduce the risk of poor deployment decisions, not eliminate it. Results are conditioned on the tested population, time window, metric choices, and sample size, and may not generalize. Confounding, insufficient duration, novelty effects, and metric misspecification can all lead to misleading conclusions.

Best practices

Define the hypothesis, primary outcome metric(s), and success criteria before running the test, so the analysis is not shaped retroactively by the observed data.

Predetermine sample size and test duration based on the effect size worth detecting, and avoid stopping the test early on the basis of interim results unless using a method designed for sequential analysis.

Verify that randomized assignment produced comparable groups and that treatment and control differ only in the variable under test, documenting any imbalances or confounders.

Treat A/B results as one input alongside independent validation, monitoring, and governance review rather than as a stand-alone approval, and record where the test's scope leaves risks unmeasured.

Examine outcomes beyond the primary metric, including performance across relevant subgroups and any operational or risk-relevant signals, to surface effects the headline metric may obscure.

State the limits of generalizability explicitly, noting that conclusions apply to the tested population, time period, and conditions and should be re-evaluated when the deployment context changes.