Definitions and comparisons
Data lineage vs data provenance: the difference, with a worked example
By SourceX Editorial · Reviewed by Noah Loul ·
Short answer
Data lineage traces how data moves and changes inside your pipelines: which job read which table, what it did, and what it wrote. Data provenance is the fuller record of where data came from and why it can be used: its origin, collection context, rights, transformations and custody. Lineage is one input to provenance. A buyer of your dataset needs both, and lineage tools alone will not produce the provenance part.
Key takeaways
- Data lineage records how data moved and changed through your jobs; provenance also records origin, rights, collection context and custody.
- Pipeline-level and column-level lineage from tools such as OpenLineage cover transformations well but say little about the source system or the people involved.
- For an external buyer, the gaps sit before the first job and after the last one: how the data was collected, under which terms, and who received it.
- A step table with input, transformation, output and evidence for each node turns an existing lineage graph into the backbone of a provenance record.
- System access, privacy permission, ownership and licensing rights are separate questions, and de-identification answers none of the last two.
What is the difference between data lineage and data provenance?#
Data lineage answers how a dataset was produced inside your systems; data provenance answers where it originally came from, under what conditions, and who is responsible for it at each step. Lineage is mostly technical and machine-captured. Provenance is partly technical and partly documentary.
In practice, data lineage is a graph of jobs and datasets. A dbt model reads a staging table and writes a mart; an Airflow task reads a bucket and writes Parquet files. Lineage tools record those edges, often down to the column, so an engineer can debug a broken dashboard or assess the impact of a schema change.
Provenance is broader. The W3C PROV ontology models provenance with three core types, Entity, Activity and Agent, and relations such as used, wasGeneratedBy, wasDerivedFrom, wasAttributedTo (linking an entity to an agent) and wasAssociatedWith (linking an activity to an agent). The agent part is what lineage usually lacks: who collected the data, who owned it, who approved its release, and who held it in transit.
A short way to remember it: lineage is the route the data took through your pipeline. Provenance is the route plus the passport: origin, permissions and every hand it passed through.
Data lineage vs data provenance compared#
The two overlap on transformations and diverge on everything outside the pipeline. This table is the quickest way to see which one a given question belongs to.
| Dimension | Data lineage | Data provenance |
|---|---|---|
| Question answered | Which jobs, tables and columns produced this output, and how? | Where did this data originate, under what terms, and who has been responsible for it? |
| Scope in time | From the first system your pipeline reads to the last table it writes | From original collection in the source system through delivery to a third party |
| Granularity | Pipeline, job, table and column level; sometimes row level for specific checks | Dataset and record level for origin and rights; step level for transformations and custody |
| How it is captured | Automatically, from query logs, orchestrator events and parsed SQL or code | Partly from lineage, partly from contracts, policies, notices and signed attestations |
| Typical tooling | OpenLineage events, dbt model graphs, data catalogs and warehouse lineage views | Datasheets, provenance records, rights memos, manifests with hashes, custody logs |
| Primary audience | Data engineers, analytics engineers, platform owners | Buyers, auditors, legal and privacy reviewers, model developers doing diligence |
| Reference standards | OpenLineage specification for run, job and dataset events | W3C PROV (PROV-O); Datasheets for Datasets; data provenance standards work at OASIS |
| What failure looks like | A broken dashboard nobody can trace to an upstream change | A licensed dataset nobody can show the company had the right to license |
- 01Your work records
- 02Personal details removed
- 03You approve the release
- 04Licensed to AI developers
What does data lineage capture at pipeline and column level?#
Pipeline-level lineage records which job read which datasets and wrote which outputs; column-level lineage records which input fields each output field was computed from. Most teams on a modern stack already have the first and some of the second.
OpenLineage is the common reference. Its facets pin a delivered file to the code version, SQL, schema and row counts of the run that produced it, and its column lineage maps each output field to its inputs with a DIRECT or INDIRECT type and a masking flag, plus dataset-wide dependencies such as filters. Those facets are the raw material for the provenance record below. The full event model is covered in OpenLineage for training data lineage, and the glossary has the short definition of data lineage.
Worked example: data lineage for a support-ticket dataset#
Here is one dataset traced end to end. A company is preparing three years of customer support tickets from Zendesk for an AI developer who wants to train a support assistant. The pipeline exports tickets and comments, normalizes them, redacts personal information, removes duplicates, splits train and evaluation sets, and writes delivery shards.
Each row is one node in the lineage graph. The fourth column is what a lineage tool typically records on its own. The fifth is the provenance evidence a person has to add so the step survives buyer diligence.
| Step | Input to output | Transformation | Lineage captures | Provenance evidence to add |
|---|---|---|---|---|
| 1. Source | Zendesk instance to raw export files | Export of tickets, comments and ticket fields for an agreed date range | Usually nothing; the source sits outside the pipeline | Account owner, export method and date, date range, which brands or groups were included, the customer terms and privacy notice in force during collection |
| 2. Landing | Raw export files to raw.zendesk_tickets | Load as-is into object storage or the warehouse | Run ID, job name, output dataset, row count | Checksum of each raw file, who ran the export, where raw copies are retained and for how long |
| 3. Staging | raw.zendesk_tickets to stg.tickets | Flatten comments into threads, standardize timestamps and languages, drop internal-only notes | Job, SQL or code version, column mapping for each field | Reason internal notes were dropped, field dictionary, any records excluded by policy |
| 4. Scope filter | stg.tickets to stg.tickets_in_scope | Remove tickets from accounts whose contracts restrict secondary use, and from excluded regions or products | A dataset-level dependency on the filter columns | The rule itself, who approved it, the contract or policy it reflects, count of records removed |
| 5. PII redaction | stg.tickets_in_scope to stg.tickets_redacted | Replace names, emails, phone numbers, addresses and account IDs with typed placeholders | Column lineage with a masking flag on redacted fields | Redaction method and version, entity types covered, sample-based residual check and its result, who performed it |
| 6. Deduplication | stg.tickets_redacted to stg.tickets_dedup | Drop exact and near-duplicate threads, such as macro replies and merged tickets | Job, code version, input and output row counts | Similarity threshold, what counted as a duplicate, removed count |
| 7. Train/eval split | stg.tickets_dedup to out.train and out.eval | Split by customer account so no account appears in both sets | Two output datasets and their row counts | Split key and seed, confirmation that no account crosses the split |
| 8. Delivery | out.train and out.eval to delivered shards | Write compressed JSONL shards and a manifest | Output dataset names and statistics | Manifest with file hashes, dataset version, recipient, transfer method, date, and sign-off on what was released |
What do lineage tools capture, and what must provenance documentation add?#
Lineage tools capture the transformations; provenance documentation must add origin, rights, privacy basis, human decisions and custody. Use the checklist below against your own pipeline before a buyer asks. Any row where the third column is empty is a gap a buyer's reviewer will find.
Keep four permissions separate while you fill it in: system access, privacy permission, ownership and contractual licensing rights. De-identification addresses privacy risk; it does not by itself create a right to license. The full framework is in data privacy and the right to license business records.
| Evidence item | Captured by lineage tools? | What provenance documentation must add | Usually supplied by |
|---|---|---|---|
| Source system and account | Partly; a dataset namespace at best | The named system, account owner, and how the export was made | Data engineering |
| Collection context | No | Why the data was originally collected, who it describes, and the notices or terms in force at the time | Product, legal, privacy |
| Ownership and rights basis | No | Who holds the data, any third-party content inside it, and the basis for licensing it to this buyer | Legal |
| Contract restrictions | No | Customer or vendor terms that limit secondary use, and the filter that enforces them | Legal, data engineering |
| Privacy basis and de-identification | Partly; a masking flag on columns | Method, entity types, residual-risk check and who performed it | Privacy, data engineering |
| Transformations | Yes; jobs, code versions, column mappings | Plain-language reasons for each exclusion or change | Data engineering |
| Volumes at each step | Yes; row counts and statistics | Reconciliation from source count to delivered count | Data engineering |
| Delivered version | Partly; dataset version facets | A manifest with file hashes tied to the run that produced them | Data engineering |
| Custody and handoff | No | Who held the data, where, and how it moved to the recipient | Security, operations |
| Permitted use | No | The intended and prohibited uses agreed with the buyer | Legal, commercial |
What provenance will an external buyer ask for?#
Buyer diligence questions usually follow the Datasheets for Datasets template, whose paper groups questions into seven categories: motivation, composition, collection process, preprocessing, distribution, maintenance, and legal and ethical considerations. Your lineage graph answers some of them; the rest need documents. The full field lists are in Data Provenance Standards explained for data suppliers and the provenance standards overview.
| Datasheets category | Where the answer comes from |
|---|---|
| Motivation and composition | Product and data owners, not the pipeline |
| Collection process and consent | Source-system records, notices and legal; no lineage tool captures them |
| Preprocessing, including whether raw data was saved | The lineage graph and job code |
| Distribution and maintenance | Delivery manifest, custody log and release terms |
| Legal and ethical considerations | Rights memo and privacy record |
How do you turn the lineage you already have into a provenance record?#
Start from the lineage graph for the exact runs that produced the delivery, then attach the non-technical evidence to each node. The steps below work whether your lineage lives in OpenLineage events, a catalog or dbt docs.
- Freeze the delivery version. Record the run IDs that produced the delivered files and the code version of each job, so the graph you document is the graph that ran.
- Export the graph for those runs only. Pipeline-level edges plus column lineage for any field that was masked, filtered or derived.
- Map it to provenance terms. Treat each dataset as an entity, each run as an activity, and each owner, approver or processor as an agent, following the W3C PROV model. Add the agents the lineage tool never saw: the source system owner, the person who approved release, the processor that redacted data.
- Add the source node. Write down the system, account, date range, export method and the terms and notices in force when the data was collected.
- Attach the rights memo. A short note from counsel on ownership, third-party content and contract restrictions, and the filter rules that enforce them.
- Reconcile counts. Show the record count at each step from export to delivery, with a reason for every drop.
- Write the datasheet. Answer the Datasheets questions that apply and point each processing answer at a node in the graph.
- Close with custody. A manifest of file hashes, the transfer method, the recipient and the release sign-off.
Where do lineage-to-provenance handoffs usually break?#
Most gaps sit at the edges of the pipeline: before the first job and after the last one. These are the patterns worth checking first.
- The source node is blank. Lineage starts at the raw table, so nobody recorded how the export was made or under which customer terms.
- Filters are undocumented. A WHERE clause removed restricted accounts, but the reason and the approver live only in a chat thread.
- Redaction is a flag, not a method. Column lineage says a field was masked, but nothing records which entity types were covered or how residual risk was checked.
- The graph is current, not historical. The catalog shows today's pipeline, which may differ from the run that produced last quarter's delivery.
- Custody ends at the bucket. Nobody can say who downloaded the shards or whether the delivered hashes match what was approved.
- Rights are assumed from access. Being able to export from Zendesk is treated as permission to license, which it is not.
How does this apply when you license data through SourceX?#
SourceX handles rights review, data preparation and delivery for businesses licensing operational data to AI developers, and the company approves exactly what is released, once per deal. Your lineage already documents the transformation steps; the provenance items in the checklist above are what the Evidence Packet is built to cover, as described in the provenance metadata fields buyers now expect. The first step is a short company assessment, and no data uploads are needed to see if a company qualifies.
Frequently asked questions
Is data lineage the same as data provenance?
No. Data lineage traces how data moved and changed through your pipelines, job by job and often column by column. Data provenance also covers where the data originated, the conditions it was collected under, who had rights to it and who held it at each step. Lineage is one part of a provenance record.
What is an example of data lineage?
A support-ticket dataset is a good example: tickets exported from Zendesk land in a raw table, are flattened in staging, filtered to in-scope accounts, redacted, deduplicated and split into train and evaluation sets. Lineage records each of those jobs, their inputs and outputs, and which output columns came from which input columns.
What is the difference between data lineage and data flow?
A data flow diagram shows how data is designed to move between systems. Data lineage records how specific datasets were actually produced by specific runs, including the code version and columns involved. Lineage is evidence of what happened; a data flow is a description of the intended design.
How do you create data lineage?
Most teams capture it automatically by instrumenting their orchestrator, transformation tool or warehouse so each job emits events about the datasets it reads and writes. OpenLineage is a common open specification for those events. For a one-off dataset, a step table listing input, transformation, output and evidence for each step is a workable manual version.
What is an example of data provenance?
For the same support-ticket dataset, provenance adds the facts outside the pipeline: which Zendesk account the tickets came from, the customer terms and privacy notice in force when they were collected, which accounts were excluded by contract, how redaction was done and checked, and who approved and received the delivered files.
Does de-identified data still need provenance?
Yes. De-identification reduces privacy risk but says nothing about whether the company owns the data or may license it. A buyer will still ask where the records came from, what terms applied and what processing was done, and counsel reviews the specific facts.
Sources
- The W3C PROV ontology (PROV-O) models provenance with Entity, Activity and Agent, and defines relations including used, wasGeneratedBy, wasDerivedFrom, wasAttributedTo (linking an entity to an agent) and wasAssociatedWith (linking an activity to an agent). Source
- The OpenLineage core model consists of run events, jobs, datasets and runs; a producer must send at least a START event and a COMPLETE, FAIL or ABORT event for each run; openLineage standard facets include job source code location and version, SQL, ownership, dataset schema and version, data quality assertions, and input and output statistics such as row counts. Source
- The OpenLineage column lineage facet maps each output field to the input fields used to compute it, with DIRECT or INDIRECT transformation types, a masking flag, and dataset-level dependencies for filtering, sorting, grouping, joins and window functions. Source
- Datasheets for Datasets (Gebru et al., Communications of the ACM, December 2021) proposes that every dataset be accompanied by a datasheet documenting its motivation, composition, collection process, recommended uses and more. Source
- The Datasheets for Datasets paper groups its questions into seven categories and asks what preprocessing was done, whether raw data was saved, whether preprocessing software is available, and whether the people described consented. Source
- The OASIS Data Provenance Standards Technical Committee maintains its contributions in a public GitHub repository. Source
Related resources
See if your company qualifies
A short company assessment. No data uploads are needed.