Category: Explainability & Interpretability

SHAP Values

Also known as: SHAP, SHapley Additive exPlanations, Shapley values (in ML explainability context)

Simply put

SHAP values are a way to explain the prediction of a machine learning model by estimating how much each input feature contributed to that prediction. The approach borrows an idea from game theory, treating each feature like a player in a game and assigning it a share of the model's output based on its contribution. Positive SHAP values push a prediction higher, while negative values push it lower, helping non-experts see which inputs mattered most for a given result.

Formal definition

SHAP (SHapley Additive exPlanations) applies the Shapley value, a solution concept introduced by Shapley (1953) in cooperative game theory for fairly distributing a total payout among players according to their contributions, to attribute a model's prediction across its input features. Each feature is treated as a 'player,' and its SHAP value represents its marginal contribution to the prediction relative to a baseline, with the attributions typically summing (additively) to the difference between the prediction and that baseline. In common explanatory visualizations, each Shapley value acts as a directional force that increases (positive) or decreases (negative) the prediction, and these forces balance to produce the final output. As commonly framed, SHAP is a widely used post-hoc feature-attribution method; note that the evidence provided does not specify computational details, approximation methods, or limitations, so those aspects are out of scope for this entry.

Why it matters

SHAP values matter because many machine learning models used in regulated settings produce predictions that are difficult to explain in human terms. When a model influences a consequential decision, stakeholders often need to understand which inputs drove a particular outcome. SHAP provides a structured, feature-level attribution that can help translate an otherwise opaque prediction into a set of contributions that non-experts can inspect, supporting model documentation, review, and challenge processes.

As a post-hoc feature-attribution method, SHAP is commonly used to support explainability objectives within both AI governance and model risk management, though the two frame the need differently. Governance functions may treat explainability as part of accountability and oversight expectations, while model risk management may use feature attributions as one input to validation and ongoing monitoring. It is important not to overstate what SHAP delivers: attributions describe how a model behaves relative to a baseline, not whether the model is correct, fair, or fit for purpose. The evidence provided does not establish that SHAP satisfies any specific regulatory requirement, so its role in any given framework should be assessed in context.

Professionals should also be careful to distinguish explainability from interpretability. SHAP offers a post-hoc explanation of a model's output rather than making the model itself inherently interpretable. Using SHAP to justify a decision does not eliminate model risk or guarantee that the underlying model is reliable; it is one tool among several that can help reduce the opacity of a prediction and inform review.

Who it's relevant to

Data Scientists and Model Developers

Data scientists commonly use SHAP to examine which features contribute to individual predictions, supporting model documentation and internal review. The attributions help developers communicate model behavior to non-technical stakeholders, though developers should recognize that the evidence here does not cover SHAP's computational methods or limitations.

Model Validators and Model Risk Functions

Those responsible for validating models may use feature attributions such as SHAP as one input when assessing how a model produces its outputs. Validators should treat SHAP as supplementary evidence rather than a substitute for broader validation, since attributions describe model behavior relative to a baseline and do not by themselves confirm that a model performs appropriately.

Compliance and Governance Specialists

Governance and compliance professionals may look to explainability methods like SHAP when documenting how consequential model decisions can be explained. They should avoid assuming that the use of SHAP satisfies any particular regulatory requirement, as that determination depends on the applicable framework and is outside the scope of this entry.

Auditors and Reviewers

Auditors examining AI systems may encounter SHAP-based explanations in model documentation. Understanding that SHAP provides post-hoc, feature-level attributions—rather than a guarantee of correctness or fairness—helps reviewers scope their assessment of how well a model's decisions are explained.

Inside SHAP

Shapley value foundation

SHAP (SHapley Additive exPlanations) values are grounded in cooperative game theory's Shapley value, which allocates a payout (here, a model's prediction) among contributing players (here, input features). Each feature's SHAP value represents its contribution to moving the prediction away from a baseline or expected value.

Additive attribution

SHAP values are additive: for a given prediction, the sum of all feature SHAP values plus the baseline (expected model output) equals the model's output for that instance. This local accuracy property is central to how the method is intended to be read.

Baseline or expected value

SHAP contributions are measured relative to a reference expectation, commonly derived from a background dataset. The choice of background data materially affects the resulting attributions, so the baseline is a component that must be specified and documented.

Local versus global use

SHAP is fundamentally a local, per-instance attribution method, though practitioners often aggregate SHAP values across many instances to approximate global feature-importance summaries. The aggregated view is a derived interpretation rather than a direct property of the underlying method.

Estimation variants

Because exact Shapley computation is generally intractable for many features, implementations typically use approximations or model-specific variants (for example, tree-based or kernel-based estimators). These approximations carry their own assumptions and can affect the values reported.

Common questions

Answers to the questions practitioners most commonly ask about SHAP.

Do SHAP values reveal the true causal drivers of a model's predictions?

No. SHAP values attribute a model's output to its input features based on the model's learned behavior, not on real-world causation. They describe how a given model uses features to arrive at a prediction, not whether those features cause the outcome being modeled. Treating attributions as causal claims is a common error; they should be read as descriptions of model mechanics, not evidence of cause and effect in the underlying phenomenon.

Does producing SHAP values make a model interpretable or explainable in a compliance sense?

Not on their own. SHAP is a post-hoc attribution technique applied to an already-trained model; it does not make the model inherently interpretable, and it produces approximations whose reliability can vary by method and configuration. Whether SHAP output satisfies a given explainability expectation depends on the framework, the audience, and the use case. Professionals should distinguish post-hoc explanation from inherent interpretability and avoid presenting SHAP outputs as a complete or authoritative account of model reasoning.

When should SHAP values be computed within a model's lifecycle?

SHAP values are typically generated after a model is trained and can be applied during validation, ongoing monitoring, and individual prediction review. Their placement depends on the intended purpose, such as supporting validation activities, investigating specific decisions, or monitoring feature contribution stability over time. Because SHAP is a post-hoc technique, it does not substitute for controls applied at earlier development stages.

How do computational costs affect the use of SHAP in production?

SHAP computation can be resource-intensive depending on the method used, the model type, and the dataset size, which can constrain real-time or large-scale application. Teams often weigh exact versus approximate approaches and consider whether attributions are needed for every prediction or only for sampled or flagged cases. These trade-offs should be documented so that the chosen approach and its limitations are transparent to reviewers.

How should SHAP outputs be documented for model risk and governance purposes?

Documentation should typically record the SHAP method used, its configuration, any approximations, the reference or background data involved, and the intended interpretation of the outputs. Noting the limitations of the attributions and the contexts in which they should not be over-relied upon helps reviewers assess their appropriateness. Clear documentation supports validation and audit activities without implying the outputs are definitive.

Can SHAP values be used to assess bias or fairness in a model?

SHAP values can indicate how features contribute to predictions and may surface features associated with protected or sensitive attributes, which can inform a broader analysis. However, they measure feature attribution, not fairness itself, and should not be treated as a fairness metric. Bias and fairness assessments generally require dedicated measures and cannot be established by attribution outputs alone.

Common misconceptions

SHAP values reveal the causal effect of a feature on the outcome.

SHAP values attribute a model's prediction across its input features; they describe the model's behavior, not the real-world causal relationship between a feature and the target. Interpreting them causally is a common error, particularly when features are correlated.

SHAP provides a single, objective 'true' importance for each feature.

Attributions depend on choices such as the background/reference dataset and the estimation variant used. Different valid configurations can yield different values, so SHAP outputs should be reported alongside their assumptions rather than treated as a unique ground truth.

Producing SHAP explanations makes a model interpretable and satisfies explainability obligations.

SHAP is a post-hoc explanation technique applied to model outputs; it does not make an otherwise opaque model intrinsically interpretable. Whether it meets any given documentation, validation, or regulatory expectation depends on the applicable framework and context, which varies by jurisdiction and sector.

Best practices

Document the background/reference dataset and the specific SHAP estimation variant used, since both materially influence the resulting attributions.

Distinguish local (per-instance) explanations from aggregated global summaries in reporting, and avoid presenting aggregated SHAP importance as a direct property of individual predictions.

Avoid causal language when communicating SHAP results; describe them as attributions of the model's behavior rather than real-world cause-and-effect.

Account for feature correlation when interpreting attributions, and flag where correlated inputs may make individual feature contributions difficult to disentangle.

Treat SHAP outputs as one input to validation and review rather than as a standalone demonstration that a model is interpretable or compliant with any particular framework.

Verify that the additivity check holds for the chosen implementation (feature contributions plus baseline reconciling to the prediction) as a sanity check on the estimation used.