Category: Data Governance & Quality

Data Labelling

Also known as: Data Labeling, Data Annotation

Simply put

Data labelling is the process of attaching meaningful tags, categories, or descriptions to raw data such as images, text, or video so that a machine learning model can learn from it. For example, marking which pictures contain a cat, or tagging a piece of text as positive or negative. These labels give the raw data the context a model needs to be trained.

Formal definition

Data labelling is the activity of assigning meaningful tags, classes, or contextual annotations to raw data instances (e.g., images, text files, videos) to produce labeled datasets used for supervised or semi-supervised machine learning and deep learning. In common usage, labelling refers to the assignment of discrete tags or classes to data pieces, including simple binary tagging tasks; some sources treat 'data labelling' as a subset of the broader practice of 'data annotation,' though the terms are frequently used interchangeably and their precise scope is not consistently defined across the evidence.

Why it matters

Data labelling directly shapes what a supervised or semi-supervised model learns, because the labels supply the context that a machine learning algorithm uses to associate inputs with desired outputs. When labels are inaccurate, inconsistent, or reflect the assumptions of the people who created them, those errors propagate into the trained model's behavior. For this reason, the quality of labelled data is often treated as a foundational input to model quality rather than a peripheral preparation step.

From a model risk perspective, labelling sits upstream of the model itself and can introduce risks that are difficult to detect once a model is in production. Errors or systematic patterns embedded during labelling become part of the model's learned behavior, which is why data provenance and label quality are commonly examined during model development and validation. It is worth distinguishing here between the labelling process as a data-preparation activity and the downstream governance controls that organizations may place around it; the evidence digest describes what labelling is and how it supports learning, but does not establish specific regulatory requirements for how labelling must be governed, and none should be inferred.

The scope of the term is also not fully settled. Some sources treat data labelling as a subset of the broader practice of data annotation, focusing on straightforward tagging such as answering binary questions, while others use the two terms interchangeably. Professionals should be aware that where precise scope matters—for example, in documentation, procurement, or vendor contracts—the intended meaning of "labelling" versus "annotation" should be defined explicitly rather than assumed.

Who it's relevant to

Data Scientists and Machine Learning Engineers

Those who build supervised and semi-supervised models depend on labelled datasets as a core training input. The way data is tagged and classified affects what the model can learn, making label quality and consistency a practical concern during model development.

Model Risk Managers and Validators

Because labelling occurs upstream of model training, those examining a model's inputs may review the provenance and quality of labelled data as part of understanding sources of model risk. Note that validation of a model and verification of its data inputs are distinct activities; label quality is one contributor to a broader assessment and not a substitute for it.

Data Operations and Annotation Teams

Teams responsible for producing labelled data need clarity on task definitions, particularly given that 'labelling' and 'annotation' are used inconsistently across sources. Explicit conventions help ensure that tags and classes are applied uniformly across a dataset.

AI Governance and Procurement Specialists

Those overseeing AI systems or contracting labelling services may need to define the intended scope of terms like labelling and annotation in documentation and vendor agreements, since the evidence shows these terms lack a single consistent definition. This is a matter of clear specification rather than a settled regulatory requirement.

Inside Data Labelling

Annotation Task Definition

The specification of what labellers are asked to produce, including label categories, decision rules, and edge-case handling. In many workflows this is captured in labelling guidelines or an annotation schema that aims to make labelling decisions consistent across annotators.

Ground Truth Construction

The process of establishing reference labels treated as correct for training and evaluation. It is important to note that labels are typically an approximation of a target concept rather than an infallible truth, and their quality directly affects downstream model performance and evaluation validity.

Labeller Workforce and Sourcing

The people or systems producing labels, which may include in-house experts, crowdsourced annotators, third-party vendors, or model-assisted (semi-automated) labelling. The choice affects domain expertise, cost, and the governance controls needed over data handling.

Quality Assurance Mechanisms

Controls such as inter-annotator agreement measurement, adjudication of disagreements, gold-standard checks, and reviewer sampling used to assess and improve label reliability. These measure consistency and, where a reference exists, accuracy of labels.

Provenance and Documentation

Records of how labels were generated, by whom, under which guideline version, and with what tooling. Such documentation supports traceability, reproducibility, and review by second-line or independent functions, and is commonly relevant to model risk and governance evidence.

Data Handling and Compliance Considerations

Controls over the data being labelled, which may involve personal, sensitive, or regulated information. Depending on jurisdiction and use case, this can raise privacy, confidentiality, and consent considerations that fall within broader governance obligations.

Common questions

Answers to the questions practitioners most commonly ask about Data Labelling.

Is data labelling just a one-time task completed before model training?

No. While labelling is often concentrated in the pre-training phase, treating it as a single one-time task is a common misconception. In many practices, labelling is revisited across the model lifecycle—for example, when data distributions shift, when label definitions are refined, or when new data is collected for retraining or monitoring. Labelling is more accurately understood as a recurring activity tied to ongoing model maintenance rather than a discrete step that ends once training begins.

Does having labelled data guarantee that a model will perform well or be free of bias?

No. High-quality labels are typically a contributing factor to model performance, but they do not by themselves guarantee good performance or the absence of bias. Labelling captures the judgments, assumptions, and potential errors of the labellers and the labelling schema, which can introduce or reinforce bias rather than remove it. Conflating well-labelled data with a well-performing or fair model is a frequent error; label quality is one input among many and should not be treated as a sufficient condition for reliable outcomes.

How can labelling consistency be assessed when multiple labellers work on the same dataset?

Consistency across labellers is commonly assessed by having more than one labeller annotate overlapping samples and comparing their agreement. Where labellers disagree, teams often review the cases to determine whether the disagreement stems from ambiguous instructions, an unclear labelling schema, or genuine edge cases. The results can inform revisions to labelling guidelines. The appropriate approach depends on the task, the volume of data, and the tolerance for label error in the intended use.

What should labelling guidelines typically document?

Labelling guidelines commonly document the definitions of each label, examples and counter-examples, instructions for handling ambiguous or edge cases, and the process for escalating uncertain items. Clear documentation helps reduce inconsistency between labellers and supports later review of how labels were assigned. The level of detail that is appropriate varies with the complexity of the task and the risk associated with the model's intended use.

How can labelling decisions be made traceable for later review or audit?

Traceability is typically supported by recording who applied a label, when, under which version of the labelling guidelines, and the rationale for contested or ambiguous cases. Retaining a record of guideline versions and any changes to label definitions allows reviewers to understand how and why labels were assigned at a given point. The extent of documentation retained often depends on the governance expectations and the level of scrutiny the model is expected to receive.

What are practical options for handling ambiguous or edge-case items during labelling?

Common options include defining an explicit category for uncertain or out-of-scope items, routing ambiguous cases to a more experienced reviewer, or flagging them for adjudication rather than forcing a label. Which approach is suitable depends on how the labels will be used and how costly a mislabelled edge case would be. Documenting how such cases were resolved helps maintain consistency and supports later review.

Common misconceptions

Labelled data is objectively correct 'ground truth.'

Labels are typically an approximation of a target concept and can embed annotator judgment, guideline ambiguity, and systematic error. The term 'ground truth' reflects a reference standard used in practice, not a guarantee of correctness.

Measuring label accuracy is the same as validating the model.

Label quality assessment concerns the reliability of the input data, while model validation (as commonly distinguished from data quality checks) evaluates whether the model is conceptually sound and fit for its intended use. Poor labels can undermine both model performance and the validity of validation results, but assessing labels does not by itself constitute model validation.

High inter-annotator agreement means labels are unbiased and fair.

Agreement measures consistency among annotators, not the absence of bias in the underlying labels. Annotators can consistently apply a guideline that itself encodes systematic bias, so consistency and fairness are distinct properties that should be assessed separately.

Best practices

Maintain versioned, written labelling guidelines with explicit decision rules and documented handling of edge cases, and treat guideline changes as controlled changes.

Assess label reliability using appropriate quality measures such as inter-annotator agreement and adjudication of disagreements, and record the results as governance evidence.

Document provenance for labels, including who produced them, under which guideline version, and using what tooling, to support traceability and independent review.

Separate the assessment of label consistency from the assessment of potential bias in the labels themselves, and evaluate each explicitly rather than assuming agreement implies fairness.

Apply data handling and privacy controls appropriate to the sensitivity of the data being labelled and the applicable jurisdiction, rather than assuming labelling is out of scope for compliance.

Where model-assisted or semi-automated labelling is used, retain human review of a defined sample so that automation does not silently propagate systematic errors into the ground truth.