OpenLineage for Training Data: Recording Lineage from Delivery to Model

This page covers the standard and its instrumentation. For the concepts, see the glossary entries on data lineage and data provenance; for the compliance goal of answering "which models used this dataset", see tracing which models trained on which licensed dataset.

The OpenLineage model: jobs, runs, datasets and facets

OpenLineage describes lineage with three core entities, jobs, runs and datasets, plus extensible metadata blocks called facets [1]. It is catalogued as an open metadata standard rather than a single vendor's format [6]. A job is a process definition such as curate_support_tickets, a run is one execution of it identified by a UUID, and datasets are the inputs and outputs that run touched [1][2]. A client posts a RunEvent with an eventType (typically START and then COMPLETE or FAIL), the job's namespace and name, the run.runId, and inputs and outputs arrays [2].

Datasets are identified by a namespace (the storage system, such as s3://ml-landing or a warehouse connection) plus a name (the path or table). That pair is the join key across your whole graph, so naming discipline matters more than any tool choice. If ingest writes s3://ml-landing/vendor-x/2026-09/ and curation reads s3://ml-landing/vendor-x/2026-09 without the trailing slash, many backends will draw two disconnected nodes.

Facets hang off each entity [2][3]:

• Dataset facets describe the data: schema, data source, column-level lineage, dataset version, ownership and documentation. Spark integrations commonly emit a columnLineage facet.
• Input and output facets describe what a specific run did to a dataset, such as row counts or bytes written.
• Run facets describe one execution: parent run, nominal time, error message, or integration-specific arguments such as airflow_runArgs.
• Job facets describe the job definition: source code location, SQL text, job type.

Every facet carries _producer (a URI for the code that produced it, ideally a tagged release or git SHA) and _schemaURL (a JSON Schema reference for that facet version) [2]. Some backends reject custom facets that omit either field, so test against yours.

Where lineage breaks in an ML training pipeline

Lineage in ML pipelines usually breaks at the training step, not in the warehouse. Airflow, Spark and dbt have mature emitters; PyTorch training loops, Hugging Face Trainer scripts and ad hoc notebook preprocessing do not emit OpenLineage events on their own, so the graph stops at "tokenized shards written" and never reaches "checkpoint produced".

A typical licensed-data path has five hops, and each needs an emitter:

1. Delivery landing. The supplier's files land in your bucket or table (see cross-account cloud bucket delivery). Nothing runs here unless you add a small "register delivery" job that emits an output event for the landed snapshot.
2. Validation. Manifest and checksum checks (see dataset manifests and checksums) should run as a job whose input is the landed snapshot and whose output is the accepted snapshot.
3. Curation. Deduplication, filtering, PII scans and joins, often in Spark or dbt.
4. Packaging. Tokenization, sharding into Parquet, Arrow or WebDataset files, and train/eval splits.
5. Training and evaluation. The run that consumes shards and produces a checkpoint, and the eval runs that consume the checkpoint.

The failure modes are predictable. Copies made with aws s3 sync or gsutil cp outside an orchestrator leave no event. Notebook steps produce orphan datasets. Training jobs that read a glob such as shards/*.parquet report nothing unless you resolve the glob to a versioned dataset name before emitting.

Instrumenting Airflow, Spark and dbt

Most pipeline engines can emit OpenLineage without code changes; you configure a transport and a namespace. In Airflow, the apache-airflow-providers-openlineage package [5] extracts lineage from supported operators and sends it to the configured transport. Spark is a separate component: enabling the Airflow provider does not by itself instrument the Spark jobs it submits, so the OpenLineage Spark integration must also be installed and enabled in the Spark application.

For Spark, the integration implements a SparkListener. You add the OpenLineage Spark package and set spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener, then point the HTTP transport at your backend and set spark.openlineage.namespace. Configuration keys have changed across releases, so check them against the docs for the integration version you deploy.

For dbt, the dbt-ol wrapper runs dbt and builds events from its artifacts. run_results.json records timing and status for each node that executed in the invocation, and models outside the invocation's selection do not appear [7]. That means a model excluded from a run produces no lineage for that run, which is correct but can surprise auditors who expect every model listed.

For Python steps and training code, use the OpenLineage Python client directly: construct a RunEvent, attach facets, and emit through an HTTP, Kafka, file or console transport [4]. Wrap emission in a small context manager so every training script emits START before reading data and COMPLETE or FAIL after writing the checkpoint.

A custom license facet for licensed datasets

The standard facets record schema and versions, but not commercial terms, so license metadata belongs in a custom dataset facet. OpenLineage lets you extend the base facets and carry your own facets in the event payload [3]. The spec asks you to use a distinct project prefix so your facet cannot collide with standard ones, to name the schema type with the {prefix}{name}{entity}Facet pattern in PascalCase, and to publish an immutable, canonical _schemaURL pinned to a git SHA or tag rather than a branch; confirm these rules against the current spec before you freeze a schema.

Illustrative example: invented to show structure; it does not describe an available dataset.

{
  "eventType": "COMPLETE",
  "eventTime": "2026-09-14T18:22:05Z",
  "producer": "https://git.example.com/ml-platform/lineage-emitters/tree/v1.4.2",
  "schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/definitions/RunEvent",
  "run": { "runId": "0192d6f1-7a3c-7e2b-9c41-5b0f3e8a2d17" },
  "job": { "namespace": "ml-platform", "name": "curate.support_tickets_v3" },
  "inputs": [{
    "namespace": "s3://ml-landing",
    "name": "deliveries/lic-2026-0412/d003",
    "facets": {
      "acme_license": {
        "_producer": "https://git.example.com/ml-platform/lineage-emitters/tree/v1.4.2",
        "_schemaURL": "https://git.example.com/ml-platform/facets/blob/3f9c2ab/AcmeLicenseDatasetFacet.json",
        "licenseId": "LIC-2026-0412",
        "deliveryId": "D003",
        "datasetVersion": "2026-09-01",
        "permittedUses": ["fine_tuning", "evaluation"],
        "prohibitedUses": ["redistribution"],
        "deidentificationMethod": "pseudonymized-tokens-v2",
        "termEndsOn": "2028-08-31"
      }
    }
  }],
  "outputs": [{
    "namespace": "s3://ml-curated",
    "name": "support_tickets/v3/2026-09-14"
  }]
}

Three design choices make this facet useful later. First, licenseId and deliveryId should match the IDs in your contract register and delivery manifest exactly, so a query can join lineage to legal records. Second, emit the facet on the landing dataset and propagate it: downstream jobs should copy acme_license onto their outputs, or a lineage query has to walk the graph backward every time. Third, keep the facet free of personal data and contract text; reference the agreement, do not paste it.

Before you roll the facet out, check these against your design:

• Facet key uses your project prefix; schema type name follows {prefix}{name}{entity}Facet.
• _schemaURL is pinned to a commit SHA or tag, and _producer points to a tagged emitter release.
• licenseId, deliveryId and datasetVersion match the contract register and delivery manifest.
• Permitted and prohibited uses use a controlled vocabulary your policy engine can evaluate.
• Every job that writes a derived dataset copies the facet to its outputs.
• Training and eval runs list resolved shard datasets as inputs, not globs.
• A test fails the pipeline if a training run's inputs lack a license facet.

Marquez and other lineage backends

Marquez is the OpenLineage project's reference metadata server, and it is the quickest way to see whether your events form a connected graph. It stores events received on its lineage API endpoint (/api/v1/lineage in common Spark examples) and renders jobs, datasets and runs in a web UI. Because it rejects malformed events, it can also serve as a basic check in CI: post sample events from each emitter and fail the build on rejection.

In production, many teams send the same events to a catalog that ingests OpenLineage, such as a cloud data catalog that maps OpenLineage facets to its own lineage model [3]. Whatever the backend, keep raw events: write them to an append-only bucket through the file or Kafka transport as well as the HTTP endpoint. Backends change; the event log is what you can replay into the next one.

Linking lineage to versions, model cards and governance records

Lineage events become evidence only when they join to dataset versions, model registry entries and documentation. Record the training run's runId in the model registry entry for each checkpoint, and record the model name and version as an output dataset or custom run facet on that training run. Then "which checkpoints consumed delivery D003" is a single graph query, and so is the reverse.

Pair lineage with stable dataset versioning (see versioning licensed datasets) so that datasetVersion points to an immutable snapshot rather than a mutable path. For incremental purchases, each delivery should be its own dataset version, which makes incremental deliveries versus full refreshes a lineage design decision as well as a transfer decision.

On the governance side, lineage supports several obligations without satisfying them by itself. Article 10 of the EU AI Act requires high-risk systems to be developed on training, validation and testing data subject to data governance and management practices [8]; as of October 2026, Regulation (EU) 2026/1744 has reportedly moved the high-risk application dates to 2 December 2027 for Annex III systems. Dataset documentation formats such as Croissant-RAI cover data life cycle metadata at the dataset level [9], and NIST SP 800-218A adds practices on the integrity of training and fine-tuning data [10]. OpenLineage supplies the run-level evidence those documents point to.

Operating lineage after delivery

Lineage needs owners, retention and tests, or it decays within a quarter. Assign each emitter to the team that owns the job, version the facet schemas in a repository with code review, and alert when a training run completes without input datasets. When a license ends or a supplier asks for deletion, the same graph lists every derived dataset and checkpoint to review, which links lineage to access controls for licensed training data and to your retention schedule.

Agree the identifiers with suppliers before the first delivery. A technical delivery specification that fixes the delivery ID format, the snapshot path convention and the manifest fields lets your landing job emit correct lineage automatically. More delivery patterns are collected in the schemas, packaging and delivery hub and the wider AI data buyer guides.

If you are sourcing the data itself, describe the dataset you need to SourceX so licensing terms are agreed before your pipeline sees a file.

Sourcing licensed training data you can trace

SourceX sources operational datasets from US companies on request and manages the commercial process, including licensing and ongoing purchases. Every dataset is rights-reviewed and delivered under a license defining records, uses, term and delivery, through private, access-controlled workflows after an executed agreement and supplier approval. Describe the data you need at sourcex.si/buyers.