Text Annotation Formats: BIO Tags, Standoff Spans and Web Annotation

Why the encoding choice matters more for text than for images

Text labels fail silently because the anchor, the string itself, changes during ordinary preprocessing. An image bounding box survives a file copy; a character span does not survive Unicode normalization, whitespace collapsing, HTML stripping, PII redaction or re-tokenization unless the format records exactly which string it indexes. A span that drifts by two characters still parses, still loads into a training loop and still produces a plausible-looking entity, so the defect only shows up as degraded F1 or as entities that straddle word boundaries.

The three families covered here sit at different points on that robustness curve. Inline token tags (BIO, IOB2, BIOES, CoNLL columns) encode labels as a property of tokens. Standoff formats (brat .ann, JSON span lists, spaCy-style (start, end, label) tuples) encode labels as ranges over a character sequence. The W3C Web Annotation model wraps a range in a target and selector structure designed for interchange. For packaging context, see the dataset delivery formats hub and the companion guide on delivering original files with sidecar metadata.

BIO, IOB2 and CoNLL columns: tokenizer-bound by design

Inline tags are the most convenient training format and the least portable delivery format, because every label is attached to a token boundary chosen by one specific tokenizer. In IOB2 (the variant most people mean by "BIO"), each token gets B-TYPE at the start of an entity, I-TYPE inside it and O outside; BIOES/BILOU adds end and single-token markers. CoNLL-style files put one token per line, with whitespace-separated columns (token, sometimes POS or chunk tags, then the NER tag) and a blank line between sentences.

The failure modes are predictable:

• Tokenizer mismatch. If the supplier split "U.S.-based" into three tokens and your pipeline splits it into five, there is no lossless way to re-align the tags without the original character positions.
• Lost whitespace. One-token-per-line files usually discard the original spacing, so you cannot reconstruct the source string, and you cannot join the labels back to the document of record.
• Scheme ambiguity. Original IOB1 uses B- only when two same-type entities are adjacent, while IOB2 starts every entity with B-. A file that does not declare its scheme will be scored differently by strict and lenient evaluators.
• Nested and discontinuous entities. A flat tag sequence cannot express "Bank of [America]" where both the organization and the location are labeled, or a split mention such as "left and right [ventricle]".
• Subword training. When you fine-tune a transformer, word-level tags must be projected onto WordPiece or BPE pieces anyway. Fast tokenizers return an offset_mapping of character ranges per piece, which is far easier to use if the labels are already character ranges.

Accept CoNLL or IOB2 only as a convenience export generated from standoff spans, with the tokenizer name and version recorded, and with a script that regenerates it.

Standoff spans: offsets over an exact text you also receive

Standoff annotation is the robust default, because labels live outside the text and point into it by character range. The brat format is the reference example: each .ann file sits beside a .txt file with the same base name, the tool never modifies the text, and offsets are 0-based with the start character included and the end character excluded, so "Japan" at the start of a document is 0 5. A text-bound line carries an ID, a type, start and end offsets and the covered text; discontinuous spans use semicolon-separated ranges, and relations reference other annotation IDs rather than offsets.

JSON or JSONL span lists follow the same idea and are easier to load at scale: one record per document with the text (or a pointer to it), a spans array of start, end, label and text, and optional relations. The covered-text field is a cheap but powerful integrity check, since text[start:end] == span.text must hold for every span.

Two conventions must be stated explicitly in the delivery spec, because tools disagree:

1. Offset unit. Python str slicing counts Unicode code points; JavaScript and Java strings count UTF-16 code units, so an emoji or a character outside the Basic Multilingual Plane counts as two; Rust and many C libraries count UTF-8 bytes. A file produced in a browser-based annotation tool and consumed in Python will be off by one for every astral-plane character that precedes a span.
2. Interval convention. Half-open [start, end) is the common choice in brat and most JSON schemas. Inclusive end offsets exist in older tools and some XML formats, so ask, do not assume.

Normalization and redaction: deliver the string the offsets index

Offsets are only valid against the exact byte sequence they were computed on, so the delivery must include that sequence and a hash of it. Unicode defines NFC, NFD, NFKC and NFKD normalization forms, under which visually identical text can have different code point sequences [1]. If a supplier annotates NFD text (where "é" is two code points) and you normalize to NFC on ingest, every span after the first accented character shifts left. NFKC is worse for this purpose because it also folds compatibility characters such as ligatures and full-width forms.

Redaction is the other common cause of drift. Replacing "Jane Smith" with [NAME] changes the length of the string, so spans computed before redaction are wrong afterward. A correct pipeline either annotates the redacted text, or recomputes offsets after each substitution and records the replacement map. Because personal details in operational text (support tickets, emails, contracts) are typically removed or replaced before delivery, the buyer question is simply: which version of the text do these offsets point to? For the redaction techniques themselves, see LLM-based PII redaction compared with NER and regex.

Other preprocessing that silently invalidates offsets: converting CRLF to LF, stripping a UTF-8 byte order mark, trimming trailing whitespace, unescaping HTML entities (& becomes &), and extracting text from PDF or HTML with a different extractor than the annotator used. The rule is simple: never let anyone re-derive the text; ship it.

W3C Web Annotation: selectors for interoperable labels

The W3C Web Annotation model is the interchange option when labels must be shared across tools, linked to documents by URI, or carry provenance about who labeled what. It is published as a W3C Recommendation together with a companion vocabulary that defines its RDF terms. An annotation has a body (the label, for example a TextualBody with purpose: "tagging") and a target whose source identifies the document and whose selector identifies the span.

Two selectors matter for NER. TextPositionSelector gives start and end positions counted from 0, with the end character excluded, which is the same convention as brat. TextQuoteSelector gives the exact selected text plus prefix and suffix context, so a consumer can re-anchor the span by search if the document changed slightly. Shipping both on each target is a practical redundant pattern: positions for speed, quotes for recovery and verification. W3C also published the selectors separately as the Selectors and States Working Group Note.

The trade-off is verbosity. A JSON-LD annotation per entity is much larger than a brat line, so many teams keep compact JSONL spans for training and generate Web Annotation documents for exchange or audit.

Comparing the three families for a delivery spec

The decision is mostly about where the source of truth lives and how many tools will touch it.

Illustrative delivery record and acceptance checks

A good span delivery couples one JSONL record per document with a manifest that pins the text, the encoding and the label schema.

Illustrative example: invented to show structure; it does not describe an available dataset.

{
  "doc_id": "tkt-000412",
  "text_file": "text/tkt-000412.txt",
  "text_sha256": "<sha256 of the exact UTF-8 bytes>",
  "text_encoding": "utf-8",
  "unicode_normalization": "NFC",
  "newline": "LF",
  "offset_unit": "unicode_code_point",
  "interval": "half_open",
  "redaction_applied_before_annotation": true,
  "label_schema": "labels/v3.json",
  "spans": [
    {"id": "T1", "start": 27, "end": 41, "label": "PRODUCT", "text": "Router X200 v2"},
    {"id": "T2", "start": 58, "end": 64, "label": "PERSON_PLACEHOLDER", "text": "[NAME]"}
  ],
  "relations": [{"id": "R1", "type": "REPORTED_BY", "arg1": "T1", "arg2": "T2"}],
  "annotator_ids": ["a07", "a12"],
  "adjudicated": true
}

Acceptance checklist to run before you sign off:

• Recompute text_sha256 for every text file and compare with the manifest (see verifying a delivery with manifests and checksums).
• Assert text[start:end] == span.text for 100% of spans, using the declared offset unit; any mismatch rate above zero means a unit or normalization error, not annotator noise.
• Check that no span starts or ends in the middle of a surrogate pair or combining sequence.
• Confirm every label appears in the label schema, and every relation argument resolves to an existing span ID.
• Regenerate IOB2 from the spans with your own tokenizer and count spans that do not align to token boundaries; review a sample.
• Re-score inter-annotator agreement on a sample, as described in the annotation quality audit guide.

Documentation of how labels were produced (guidelines version, annotator pool, adjudication) can also be expressed in machine-readable form; Croissant-RAI lists data labeling among its documentation use cases [2], and the Croissant metadata guide covers what to request.

What to put in the request when buying labeled text

The most useful request specifies the format of the source of truth, not just the label set. Name the entity types and their definitions (see the named entity recognition glossary entry), whether nesting and discontinuous mentions are in scope, the offset unit and interval convention, the normalization form, whether annotation happened before or after redaction, and which derived views (IOB2 for a named tokenizer, Web Annotation JSON-LD) you want generated. Ask for the annotation guidelines and the adjudication policy as part of the diligence pack.

SourceX sources operational datasets from US companies, including support and sales histories, engineering records, documents, and finance and legal workflows, on request rather than from stock. Every dataset is rights-reviewed for ownership and consents, personal details such as names, emails, phones and account numbers are removed or replaced before delivery with the method recorded, and delivery runs through private, access-controlled workflows. If you already have a span specification like the one above, you can describe the labeled text data you need to SourceX; for annotation-heavy requests, the expert annotations and labels page explains the category.

Sourcing labeled text for NER and extraction

If your NER, extraction or evaluation work needs labeled operational text from real businesses, describe the data, its span format and its intended use, and SourceX looks for US companies that hold it. Nothing is contracted until a supplier agrees, and every release is approved by the supplying company. Start your request at sourcex.si/buyers.