Document Annotation Schema: One Spec for Layout, Reading Order, Tables and Fields

This page is for the person who writes the instructions, whether the annotators are in-house, at a vendor, or at the company that will license you pre-labeled documents. For the wider map of tasks and datasets, start at the Document AI data hub. For the basic term, see the data annotation glossary entry.

Why a single flat label list fails for documents

A flat label list fails because document tasks depend on each other, and one mixed list hides where disagreement comes from. If "Table" is a region label and "Total amount" is a field label in the same palette, an annotator who draws a table box slightly too tight also cuts off the field it contains. Nobody can tell whether the error belongs to layout or extraction. Layering fixes this: each layer refers to IDs from the layer below, so a reviewer can freeze layout and audit only links.

Public datasets show the cost of underspecifying a layer. DocLayNet has 80,863 manually annotated pages in 11 layout classes, delivered in COCO format. Its authors annotated a subset two or three times to measure inter-annotator agreement, and baseline detectors scored roughly 10% below that agreement [1]. The PubTables-1M authors name ambiguity and inconsistency in table annotations as a core problem. They had to add a canonicalization and quality-control procedure to get consistent ground truth [4]. Both lessons carry over to private business documents, where layouts vary more than in scientific papers.

The five layers and what each one stores

Each layer stores one kind of decision, and it points to the layer below by ID instead of repeating geometry. The table below is a starting schema to adapt. It keeps detector-style labels (boxes), sequence labels (order and tokens) and graph labels (links) separate.

Illustrative example: invented to show structure; it does not describe an available dataset.

Two design choices matter more than the label names. First, coordinates live only in L1 and token tables, and every other layer references IDs, so a corrected box propagates everywhere. Second, the L1 classes should be a superset you can collapse. If you keep DocLayNet's 11 classes as a subset and add business-specific ones such as Stamp and Signature, you can still benchmark against public detectors [1]. For a deeper look at public layout taxonomies, see layout analysis datasets beyond DocLayNet and PubLayNet.

Edge-case rules annotators actually need

Edge-case rules should be written as "if you see X, label Y, because Z" with a screenshot, since prose definitions alone do not settle borderline pages. Collect cases during a pilot round and add each disputed page to the guideline as a numbered example. The cases below come up in almost every business-document project.

• Stamps and signatures over text. Draw the Stamp region over the full stamp, and keep the underlying Text region intact. Mark overlapping tokens with occluded: true instead of deleting them. Never merge a stamp into the text block.
• Multi-column flow. Reading order follows the column, not the visual row. A full-width heading that interrupts two columns starts a new flow group. Sidebars and pull quotes go in a separate flow_group, so you can score main-text order on its own. The reading order data guide covers flow rules in more depth.
• Tables spanning pages. Label each page fragment as its own Table region and set continues_from to the previous fragment's table_id. Repeated header rows on page two are labeled is_header: true and flagged repeated_header, so merge logic can drop them. See multi-page and long document data for packet-level issues.
• Values without printed keys. An amount in a box with no label is still a value. Link it to an implicit key by setting key_text: null and field_type: total_due, and do not invent a key region. FUNSD-style key-value linking assumes printed keys, so this rule has to be explicit [2].
• Handwritten corrections. Label both the struck-through printed value and the handwritten value. Set value_normalized from the handwriting only if the guideline says the correction governs.
• Empty cells. Annotate empty table cells explicitly. PubTabNet's authors note that an older table metric ignored empty cells and global structure, which is one reason TEDS scores the whole HTML tree [3].

Key-value linking: entities, relations and normalization

Key-value linking should be labeled as a graph, where entities are nodes and typed links are edges, rather than as paired text strings. FUNSD annotates noisy scanned forms for entity labeling and entity linking, which makes it a useful public reference for the graph shape [2]. Production specs usually need more than a question-to-answer link. They also need header-to-key grouping, value-to-unit links, and line-item rows that tie quantity, unit price and amount together.

Specify three things per field. The first is the raw span exactly as printed, including OCR errors. The second is the normalized value in a fixed format, such as ISO 8601 dates, decimal amounts with a currency code, and digits-only tax IDs. The third is the link set. Keeping raw and normalized values apart lets you train extraction models on the raw text and score them on normalized values. Also state whether annotators label every occurrence of a field or only the one the business uses. Totals that appear on both page one and the last page are the classic source of disagreement.

Agreement metrics and adjudication per layer

Each layer needs its own agreement metric, because one overall kappa averages away the layer that is failing. Chance-corrected statistics such as Cohen's kappa (two annotators) and Krippendorff's alpha (more annotators, missing data) suit categorical layers like page class. The right choice depends on the label type and how many raters you have [7]. Geometric and structural layers need task metrics instead.

• Regions: match boxes by IoU above a stated threshold, then report per-class F1. Report Stamp and Signature separately, because small classes get lost in micro-averages.
• Reading order: normalized edit distance between order sequences, the approach OmniDocBench uses for reading-order evaluation [6].
• Tables: TEDS treats a table as an HTML tree whose cells carry colspan, rowspan and content [3]. GriTS compares tables as grids and gives partial credit [5]. Pick one as your acceptance metric and report the other for diagnosis.
• Fields and links: exact match on normalized values, plus precision, recall and F1 on links.

Adjudication should follow written rules, not a majority vote alone. Simple voting breaks down for multi-object labels such as boxes and links, and the aggregation literature treats these tasks as distinct from simple classification [8]. Name a reviewer role per layer and log a reason code for each override. Feed recurring overrides back into the guideline as new numbered examples. Gold pages with known answers, mixed into normal queues, catch annotator drift between agreement rounds.

Versioning the guideline and the labels

Version the guideline like code, and stamp every labeled page with the version used. A change from "Stamp is optional" to "Stamp is required" silently makes older pages look like missed detections. A guideline_version field per page lets you filter, relabel or down-weight affected pages. Keep a changelog that lists the affected layers and classes for each version.

The same metadata belongs in your dataset documentation. Data Cards cover collection and annotation methods, intended use and decisions that affect model performance [10]. ISO/IEC 5259-3 sets requirements for a data quality management process for ML data but leaves the metrics to you [9]. A layered guideline with per-layer agreement targets is one concrete way to supply them.

Export mapping so one label set serves several pipelines

Map the schema to target formats up front, so moving from a detector to a token classifier or a vision-language model does not force relabeling. Store a canonical record per page, then generate the formats from it:

• Detectors: export L1 regions to COCO JSON, the format DocLayNet ships in [1].
• Token classification: join OCR tokens to L1 and L4 by geometry, then emit BIO tags plus link pairs.
• Table models: export L3 to HTML following PubTabNet-style structure for TEDS scoring [3].
• VLM and parsing models: emit Markdown or JSON in L2 reading order, with tables as HTML.

Illustrative example: invented to show structure; it does not describe an available dataset.

{"doc_id":"d-00412","page_no":2,"guideline_version":"2.3.0","page_class":"invoice","regions":[{"region_id":"r7","class":"Table","bbox":[88,410,1610,1302]},{"region_id":"r9","class":"Stamp","bbox":[1220,1350,1560,1590]}],"reading_order":{"main":["r1","r3","r7","r8"],"excluded":["r9"]},"tables":[{"table_id":"t1","region_id":"r7","continues_from":"t0","cells":[{"cell_id":"c1","row":0,"col":0,"rowspan":1,"colspan":2,"is_header":true,"repeated_header":true}]}],"entities":[{"entity_id":"e5","type":"value","field_type":"total_due","region_id":"r8","raw":"$1,2O4.50","value_normalized":{"amount":"1204.50","currency":"USD"},"key_text":null,"links":[]}],"review":{"annotator":"a17","reviewer":"q03","adjudicated_layers":["L3"]}}

Deliver records as JSON Lines: UTF-8, one valid JSON value per line, no blank lines [11]. Then validate every file against a JSON Schema before scoring. The evaluation dataset specification guide covers slices and acceptance thresholds for the held-out portion.

Using the schema when you commission or license labeled documents

The same schema works as a procurement document, because it turns "labeled invoices" into checkable acceptance criteria. Send it with your request so a vendor or a data holder can say which layers already exist in their records and which would need new labeling. Many business systems hold L0 and L4 implicitly, for example document types in a records system and posted values in an ERP. Layout and reading-order layers almost always need new work. The trade-offs are covered in annotating your own documents vs licensing pre-labeled ones and the document dataset requirements spec.

If you need real business documents to label against this schema, SourceX sources operational datasets, including documents and finance and legal workflows, from US companies on request. Every release is approved by the supplying company, and a request does not guarantee a match. You can describe the documents you need on the buyers page. For existing label sets, see licensing expert annotations and labels.

Find business documents to label against your schema

SourceX finds US businesses that hold the documents you describe and manages licensing through Find, Assess, Agree, Transact and Manage. Every dataset is rights-reviewed, has personal details removed or replaced before delivery, and ships under a license that defines records, uses, term and delivery. Start a buyer request.