Glossary
Data lineage
Quick answer
A map of where data came from and every step it passed through before reaching its current form.
Lineage tracks sources, transformations, merges and filters. It answers questions such as which system a field came from and which records were removed during cleaning.
For licensed datasets, lineage backs up the dataset documentation and helps both sides confirm that excluded records stayed excluded.
Why Data lineage matters when licensing company data to AI
Data lineage is a map of where data came from and every step it passed through before reaching its current form. It tracks sources, transformations, merges and filters, so a reviewer can see which system a field came from and which records were removed during cleaning. For a licensed dataset, lineage supports its documentation and helps both sides confirm that excluded records stayed excluded.
- 01Your work records
- 02Personal details removed
- 03You approve the release
- 04Licensed to AI developers
Related terms
What is data lineage used for?#
Data lineage is used to answer two kinds of questions: where a wrong number came from, and what will break if something upstream changes. Microsoft's Purview documentation describes the first as backward-looking work such as troubleshooting, tracing root cause in pipelines and debugging, and the second as compliance and "what if" scenarios, often called impact analysis.
The table below maps common lineage questions to the direction you trace and the people who usually ask them. The examples are illustrative.
| Use | Question it answers | Direction traced | Who usually asks |
|---|---|---|---|
| Root cause analysis | Which source or job produced this wrong value? | Upstream from the bad output | Data engineers, analysts |
| Impact analysis | Which reports, models or exports change if we alter this table or column? | Downstream from the change | Data owners, platform teams |
| Audit and compliance | Which systems and steps touched this field before it reached a report or file? | Upstream, end to end | Compliance, internal audit |
| Migration planning | What depends on the system we are retiring? | Downstream from the old system | IT and data platform leads |
How is data lineage captured?#
Lineage is captured in three main ways: by reading pipeline code, by recording jobs as they run, and by writing it down by hand. Most teams combine them, because each method sees things the others miss.
Static analysis reads SQL and transformation code before it runs. dbt, for example, says its Fusion engine fully comprehends a project's SQL, which enables precise column-level lineage. Runtime capture records what actually happened. In the OpenLineage standard, a job run must send at least a START event and a COMPLETE, FAIL or ABORT event, and the start event can carry the source code version, such as a git commit. Some platforms capture lineage natively: Databricks Unity Catalog captures runtime lineage for queries run on Databricks, down to the column level.
| Method | How it works | Strength | Typical blind spot |
|---|---|---|---|
| Static code analysis | Parses SQL or transformation code to infer inputs and outputs | Shows lineage before anything runs | Logic hidden in scripts, notebooks or manual steps |
| Runtime event capture | Jobs emit events about inputs, outputs and versions as they run | Records what actually ran, run by run | Jobs with no integration send nothing |
| Platform-native capture | The data platform records reads and writes itself | No extra instrumentation inside that platform | Stops at the platform boundary |
| Manual documentation | People record sources, exports and decisions in a register | Covers exports, approvals and offline steps | Goes stale unless someone owns it |
What breaks data lineage, and how do you check yours?#
Lineage breaks wherever data moves through a step no tool observes, so a lineage graph can look complete while missing the step that matters. Platform documentation lists concrete gaps. Databricks notes that column lineage is not captured when a source or target is referenced by path instead of table name, that lineage is not preserved for renamed tables or columns, and that inserting explicit values creates no column lineage because there is no source. Its lineage system tables also keep only a rolling one-year window.
Use this checklist before you rely on a lineage graph for an audit or an external dataset:
- Pick one delivered output and trace it back to the source system; note every step where the graph goes blank.
- Check for manual exports, spreadsheet edits and one-off scripts; record them by hand if no tool sees them.
- Look for renamed tables, schemas or columns, and link old and new names in your documentation.
- Confirm column-level lineage exists for fields that were masked, redacted or filtered, not just table-level lineage.
- Check how long your lineage tool keeps history, and export the records you need before they age out.
- Store the code version or commit for each run that produced a delivered file.
Sources for this section
- Microsoft describes lineage as supporting troubleshooting and root cause tracing, plus compliance and impact analysis. Source
- dbt's Fusion engine uses static analysis of project SQL to provide column-level lineage. Source
- The OpenLineage spec requires a START event and a COMPLETE, FAIL or ABORT event per run, and start events can carry the source code version. Source
- Databricks Unity Catalog captures runtime lineage to column level for queries run on Databricks, and does not capture column lineage for path-referenced data or preserve lineage across renames. Source
- Databricks lineage system tables keep a rolling one-year window and omit column events with no source. Source
Related resources
- IndustryHealthcare data
- IndustryConstruction data
- DataInternal knowledge bases
- DataDocument approval records
- QuestionDo AI companies buy private business data?
- QuestionCan SaaS data be licensed?
- InsightCan restaurants and hospitality groups sell their data to AI companies?
- InsightCan SaaS companies sell their data to AI companies?
- InsightSelling internal documentation to AI companies: what to know
- SolutionData partnerships between businesses and AI developers
- SolutionData licensing: granting defined rights to use your data
- GlossaryEnterprise data
See if your company qualifies
A short company assessment. No data uploads are needed.