Category: Explainability & Interpretability

Local Interpretability

Also known as: Local Explanation, Local Interpretability of Machine Learning Models

Simply put

Local interpretability refers to understanding why a machine learning model produced a specific individual prediction, rather than how the model behaves overall. For example, it explains the factors that led a model to a particular decision for one case. This is often useful when a model's full internal logic is too complex to summarize globally in a simple way.

Formal definition

Local interpretability concerns the ability to explain a model's output for a given input or prediction, as opposed to global interpretability, which characterizes the model's behavior across the input space as a whole. In the computational framing, a model is considered more locally interpretable for a given explanation type when the local form of that explanation is more readily computable. Common methods for producing local explanations of complex ('black box') models include LIME (Local Interpretable Model-Agnostic Explanations), Shapley Additive Explanations (SHAP), and integrated gradients. Empirical work has indicated that human accuracy on local interpretability tasks tends to decrease as the number of operations in the model increases, suggesting local interpretability is not absolute but varies with model complexity. Note that the distinction between interpretability (understanding the model or its reasoning) and explainability (post-hoc justification of outputs) is contested and used inconsistently across the literature; this entry treats 'local explanation' methods as one operationalization of local interpretability without asserting a single authoritative definition.

Why it matters

Local interpretability addresses a practical problem that arises when models are too complex to summarize globally in a simple way: even if the overall behavior of a system cannot be captured succinctly, stakeholders often still need to understand why a specific decision was made for a specific case. This case-level focus is frequently what matters most in operational and compliance settings, where a single adverse decision—rather than aggregate model behavior—may trigger a review, a challenge, or a requirement to justify an outcome to an affected individual.

Because local explanations aim to characterize individual predictions rather than model-wide behavior, they are often positioned as a tool for reviewing high-stakes or contested outcomes. However, local interpretability should not be treated as absolute. Empirical work cited in the evidence indicates that human accuracy on local interpretability tasks tends to decrease as the number of operations in the model increases, which suggests that the practical usefulness of a local explanation can degrade as model complexity grows. Practitioners should therefore treat local interpretability as varying with model complexity rather than as a guaranteed property.

Who it's relevant to

Data scientists and model developers

Those building and testing machine learning models use local explanation methods such as LIME, SHAP, and integrated gradients to understand why a model produced a particular prediction for an individual case, especially when the model's full internal logic is too complex to summarize globally. Awareness that local interpretability varies with model complexity is relevant when selecting explanation methods and communicating their limits.

Model validators and reviewers

Professionals reviewing individual model outputs may rely on local explanations to examine specific predictions rather than only aggregate behavior. Because evidence suggests human accuracy on local interpretability tasks can decrease as model complexity increases, validators should treat local explanations as an aid to review rather than as a definitive account of a model's reasoning.

Compliance and audit professionals

Where decisions affecting individuals must be examined or justified, local interpretability methods can help surface the factors associated with a specific decision. These professionals should be aware that the interpretability–explainability distinction is contested in the literature, and that a post-hoc local explanation is one operationalization rather than a single authoritative definition of a model's reasoning.

Inside Local Interpretability

Instance-level explanation

Local interpretability refers to understanding why a model produced a specific output for a single input or a small neighborhood of inputs, rather than characterizing the model's overall behavior. The unit of analysis is typically one prediction or decision.

Local surrogate approximation

Many local interpretability techniques approximate the model's behavior around a particular instance with a simpler, more understandable representation. Such approximations are valid only in the local region and are not intended to describe the model globally.

Feature attribution for a single prediction

A common component is the attribution of a specific prediction to the contribution of individual input features. These attributions describe the influence of features for that instance and may not generalize to other instances.

Distinction from global interpretability

Local interpretability contrasts with global interpretability, which seeks to characterize how a model behaves across its entire input space. The two operate at different scopes and answer different questions.

Relationship to interpretability versus explainability

Local interpretability sits within the broader distinction between interpretability (the degree to which a model's mechanics can be understood, often as an intrinsic property) and explainability (post hoc methods to render behavior understandable). Local methods are frequently, though not exclusively, applied as post hoc explanations.

Common questions

Answers to the questions practitioners most commonly ask about Local Interpretability.

Is local interpretability the same as global interpretability?

No. Local interpretability, as commonly defined, concerns explaining an individual prediction or a specific decision for a single input, whereas global interpretability addresses how a model behaves across the whole input space or population. A local explanation that appears intuitive for one instance does not necessarily describe the model's overall logic, and professionals frequently err by generalizing a single local explanation into a claim about the model as a whole.

Does a local explanation reveal the true internal reasoning of the model?

Not necessarily. Many local interpretability techniques produce approximations or post-hoc rationalizations of a prediction rather than a faithful account of the model's internal computation. A common pitfall is treating the output of an explanation method as ground truth about causation; it is better understood as an estimate whose fidelity should itself be assessed, especially for complex or opaque models.

When should local interpretability be applied rather than global interpretability?

Local interpretability is typically useful when the objective is to justify or contest a specific outcome—for example, explaining why a particular application was declined—or when investigating an individual case flagged during monitoring or review. Global methods are more suited to understanding model behavior overall. Many programs use both, selecting the scope according to the question being asked rather than treating one as a substitute for the other.

How can the reliability of a local explanation be evaluated?

Reliability is often examined by assessing the fidelity of the explanation to the model's actual output and its stability under small perturbations of the input. Explanations that vary substantially for near-identical inputs may indicate limited trustworthiness. Because such assessment depends on method and context, it is generally treated as an ongoing verification activity rather than a one-time check.

What role can local interpretability play in a model risk management or governance program?

Local interpretability can support activities such as reviewing contested individual decisions, documenting the basis for outcomes, and aiding investigations during monitoring. It may contribute to validation evidence and oversight, though it does not by itself constitute validation or eliminate model risk. Its use and expectations vary by sector and by the framework an organization has adopted.

What are the practical limitations to keep in mind when implementing local interpretability?

Local explanations apply to individual cases and may not aggregate into an accurate description of model behavior; some methods are approximate and can be unstable; and an explanation being understandable does not guarantee it is faithful. Implementers should scope claims to the individual prediction, document the method and its assumptions, and avoid presenting a local explanation as a complete account of how the model works.

Common misconceptions

A local explanation reveals how the model behaves overall.

Local interpretability characterizes behavior only for a single instance or its immediate neighborhood. Extrapolating a local explanation to global model behavior is a common error, since attributions and local approximations may not hold elsewhere in the input space.

A local explanation gives the true internal reasoning of the model.

Many local methods produce approximations or attributions that are consistent with the model's output for that instance; they do not necessarily reveal the model's actual internal computation. The fidelity of such approximations to the underlying model can vary and should be assessed rather than assumed.

Local interpretability and explainability mean the same thing.

Interpretability and explainability are distinct concepts that professionals typically keep separate. Local interpretability concerns understanding a specific prediction; it is often achieved through post hoc explainability techniques but is not synonymous with explainability as a whole.

Best practices

Clearly state the scope of any local explanation, noting that it applies to a specific instance or local neighborhood and should not be presented as describing the model's global behavior.

Assess and document the fidelity of local approximations to the underlying model, rather than assuming the explanation faithfully reflects the model's internal computation.

Distinguish in documentation between local interpretability (a single prediction) and global interpretability (overall model behavior), so reviewers understand what question each explanation answers.

Treat local feature attributions as instance-specific evidence, and avoid generalizing them to broader claims about model behavior without additional analysis.

Where local explanations support governance or model risk activities, record the technique used and its limitations so that downstream reviewers can weigh the explanation appropriately.