Writing Image Annotation Guidelines and Label Taxonomies for Purchased Data

Why a buyer-authored guideline beats the supplier's default labels

Labels only serve your model when the class boundaries are drawn where your model needs them, and a supplier's existing labels were drawn for some other purpose. An inspection team's "scratch" may include scuffs your model must ignore; a retailer's "out of stock" may be logged per SKU rather than per shelf facing. If you buy raw images and commission labels, or buy pre-labeled images and relabel a slice, the guideline is the only document that makes the labels mean what your loss function assumes. The tradeoff between those two routes is covered in pre-labeled vs raw image datasets.

Documentation and label files also disagree more often than buyers expect. One paper reports that SKU-110K ships bounding boxes without category labels [1], while third-party documentation describes the annotations differently [2]. Test sets of widely used benchmarks carry an estimated average label error rate of at least 3.3%, including at least 6% of the ImageNet validation set [3]. A written guideline gives you a reference to audit against instead of a README to trust.

This page covers the class semantics. Choosing the geometry (box, polygon, mask or keypoint) is a separate decision, covered in which annotation type to buy, and measuring label quality after delivery belongs to the training data quality hub.

Build the taxonomy from model decisions, not from the supplier's vocabulary

Start the taxonomy from the decisions your model must make downstream, then work back to the fewest classes that support them. If the production system only routes parts to "pass," "rework" or "scrap," a 40-class defect taxonomy is useful only if you will collapse it reliably, and every extra fine class splits already thin data. Defects are low-probability events in production, so fine-grained defect classes quickly end up with handfuls of examples [8].

Use a shallow hierarchy so coarse and fine labels stay consistent. UNSPSC, for instance, orders products in four levels from Segment down to Commodity [7]; a defect taxonomy can follow the same idea with three levels such as family (surface, geometric, assembly), class (scratch, dent, burr) and an optional attribute (severity, location zone). Annotators label at the finest level you can define unambiguously, and you derive the coarse levels by lookup rather than by asking for them twice.

Decide early which concepts are classes and which are attributes. "Corroded bolt" as a class multiplies with every component type; "bolt" with attribute corrosion: none | surface | pitting | section_loss scales. Attributes also let you add a severity scale later without re-drawing geometry.

Reserve three special values in every taxonomy:

• ignore or crowd regions that are excluded from the loss, such as dense background clutter or reflections.
• uncertain, which annotators use instead of guessing, with a required free-text reason.
• other_<family> for real objects outside the defined classes, so novel cases are logged rather than forced into the nearest class.

Write one definition card per class, with negatives and edge cases

Each class needs a card that a new annotator could apply without asking you a question. The card defines the concept in visual terms (what is visible in pixels, not what the operator knew), lists inclusion and exclusion criteria, shows at least three positive and three hard-negative images, and states the geometry rule for that class. Hard negatives matter most: the look-alike that should not be labeled teaches the boundary.

Edge-case rules belong on the card, not in a separate FAQ annotators never open. Write explicit thresholds where you can: minimum visible area in pixels, the truncation percentage at which an object is still labeled, how to handle mirror reflections, labels printed on packaging, motion blur, glare, and partial views across image tiles. Where a threshold is impossible, write a decision rule ("label if a trained inspector would call it from this image alone").

Illustrative example: invented to show structure; it does not describe an available dataset.

taxonomy_version: defects-v2.3.0
class_id: 14
name: surface.scratch
parent: surface
definition: >
  Linear surface mark with visible depth or edge contrast, length at least
  5x its width, on a painted or machined face.
include:
  - Scratches crossing a part edge (label the full visible extent)
  - Scratches under clear coat if edge contrast is visible
exclude:
  - Scuffs without a linear edge -> surface.scuff
  - Machining tool marks in a repeating pattern -> ignore
  - Marks on fixtures, gloves or background -> do not label
geometry: polygon, tight to visible edge; min visible length 20 px
occlusion: label if at least 30% of the estimated length is visible
attributes:
  severity: [cosmetic, functional]   # functional = annotator sees substrate
  zone: [A, B, C]                    # from the part drawing's cosmetic zones
uncertain_rule: use 'uncertain' with reason if glare covers the mark
examples:
  positive: [img_0412, img_0877, img_1033]
  hard_negative: [img_0219 (scuff), img_0655 (tool marks), img_0981 (reflection)]
changelog: "2.3.0: split scuff from scratch; reviewed by QA lead"

Map operational codes to model classes before labeling starts

When images come from operating records, the supplier already has labels, and they are usually operational codes rather than visual classes. Manufacturing images often carry a Material Review Board (MRB) disposition such as use-as-is, rework, repair, scrap or return to vendor; inspection programs carry finding codes; insurers carry damage or loss codes. These are decisions about the business, and the same visual defect can be scrapped on one part and accepted on another.

Write a mapping table that says, for each code, whether it can be used as a label directly, needs visual relabeling, or is metadata only. Dispositions usually become metadata or weak labels because they encode tolerance, cost and customer requirements, not appearance. Finding codes are more often near-visual, but check their definitions in the supplier's inspection procedure before you rely on them. Text fields that travel with the images, such as inspector notes, can be turned into captions; see domain captions from work records.

Illustrative example: invented to show structure; it does not describe an available dataset.

The mapping also exposes coverage gaps. If one finding code supplies most of your positives, the model learns that site's camera and lighting rather than the defect; the industrial defect image datasets page covers how buyers specify defect coverage.

Version the taxonomy and carry the version into every label file

Treat the taxonomy like an API: semantic versions, a changelog, and a version string on every annotation. A major version changes class meaning (splitting or merging classes); a minor version adds classes or attributes; a patch clarifies wording without changing decisions. Labels made under different major versions are not mixed without a migration script and a re-review sample.

Put the version where tools will keep it. In COCO JSON, the info block and the categories list travel with the file [4], so record the taxonomy version in info and keep categories[].id stable across versions; never reuse a retired ID for a new class. Add per-annotation fields for annotator_id, guideline_version and review_status if your tool supports custom attributes. At the dataset level, document labeling instructions and decisions in a Data Card [5], and if you want machine-readable records, Croissant-RAI lists data labeling among its documentation use cases [6].

Pilot, measure agreement and adjudicate before full volume

A guideline is a draft until two annotators have used it on the same images. Run a pilot of a few hundred images stratified by expected class, have at least two annotators label each independently, and compute agreement per class: Cohen's or Fleiss' kappa for image-level classes, IoU-matched F1 for boxes and masks. Classes with low agreement point to a definition problem, not an annotator problem.

Adjudicate disagreements with a named domain reviewer, add the disputed images to the class card as new examples, and bump the minor version. Then freeze a gold set of adjudicated images and seed it into production batches as hidden checks. Given measured error rates even in famous benchmarks [3], plan a post-delivery audit on a random sample regardless of the supplier's reported accuracy.

Guideline completeness checklist

Illustrative example: invented to show structure; it does not describe an available dataset.

• Model decisions listed, and every class traces to one of them
• Taxonomy tree with stable IDs, parent links and a version string
• Definition card per class: visual definition, include and exclude lists, three positives, three hard negatives
• Global rules: occlusion and truncation thresholds, minimum size, reflections, tiles, blur, glare
• Special values defined: ignore, uncertain with reason, other_<family>
• Operational code mapping table, signed off by a domain owner at the supplier
• Geometry rule per class, consistent with the annotation format you will receive
• Export format named (for example COCO JSON) with required custom fields
• Pilot plan: sample size, double-labeling rate, agreement metric and threshold per class
• Gold set and audit sample size for acceptance

What to put in the supplier request alongside the guideline

The guideline travels with your data request; it does not replace it. The request should also state the image sources you want (camera types, sites, time span), class balance targets, the export format, and the acceptance test, and should ask the supplier to confirm that the images and any existing codes can be licensed for model training. Our guide to writing a data request for suppliers and the multimodal dataset specification template cover those fields; the glossary entry on data annotation defines the terms.

Ask suppliers for a sample of real label files, not screenshots, before you sign anything, and run the sample through your own parser against the guideline [1][2]. If you need expert annotators rather than a general workforce, see licensing expert annotations and labels. For the wider image sourcing picture, start at the image data hub.

When the images you need sit in operating records at US companies, SourceX works with AI buyers to find businesses that hold them; you describe the data, and every release is approved by the supplying company. Each dataset is rights-reviewed for ownership and consents, and personal details such as names and account numbers are removed or replaced before delivery, with the method recorded and a sample checked, though no method is perfect.

Commissioning labeled images to your guideline

SourceX sources operational datasets, including engineering records, from US companies on request and manages the licensing process; categories are not inventory and a request does not guarantee a match. Bring your taxonomy and class definitions so the assessment covers both the data and its licensing permissions. Describe the labeled images you need.