Privacy and preparation
Data quality tools vs a data licensing review: what each checks
By SourceX Editorial · Reviewed by Noah Loul ·
Short answer
Data quality tools such as Great Expectations, Soda Core, dbt tests, Deequ and fg-data-profiling (formerly ydata-profiling) automatically check completeness, duplicates, validity and schema, freshness, distribution drift and referential integrity. A buyer's licensing review tests what tools cannot: rights and consent, coverage for the buyer's task, outcome label reliability, residual re-identification risk and whether a sample replays against your manifest. Run the tools first, export their results, then add the human review.
Key takeaways
- Data quality tools automate six structural checks: completeness, uniqueness, validity and schema, freshness, distribution drift and referential integrity.
- A passing test suite proves the data matches your rules, not that a buyer may use it or that it fits their task.
- A licensing review adds rights and consent, task coverage, outcome label reliability, residual re-identification risk and sample replay against the manifest.
- Export tool results with run dates, versions and the extraction query so a buyer can reproduce your numbers.
- Sequence the work: freeze, profile, assert, check joins and time, then hand reviewers an evidence pack for the human checks.
What do data quality tools actually check?#
Data quality tools check data against rules: whether values are present, unique, well formed, current, consistent over time and consistent across tables. They answer the question 'does this extract match the rules we wrote for it?' and they answer it quickly, repeatably and at full volume.
The tools fall into three working classes. Profilers scan a table and describe it: column types, missing values, duplicates, correlations and alerts. Validation and test frameworks run assertions you define and report which rows or columns fail. Monitoring and observability tools run checks on a schedule and flag when freshness, volume or distributions shift from their history.
Some well-known open source examples show the pattern. Great Expectations (GX Core) calls its Expectations unit tests for your data and can generate documentation for each set of validation results. Soda Core describes itself as a data quality and data contract verification engine, with contracts written in YAML that check missing values, invalid values, schema and row counts. dbt data tests are SQL queries that look for failing rows, and pass when none are returned. Deequ, a library built on Apache Spark, defines unit tests for data, profiles large datasets and runs anomaly detection on quality metrics over time.
None of these tools knows why the data exists, who is allowed to use it, or what model a buyer intends to train. That gap is the subject of this guide.
Which data quality tools fit which job before a licensing review?#
Pick tools by the job and the evidence they leave behind, not by brand. For a one-time licensing extract, a profiler plus a small validation suite usually covers the automated part; scheduled monitoring matters when you will deliver refreshes.
The examples below are illustrations of each class with their documented behavior, not recommendations or rankings. Many teams already have one of these in their warehouse; reuse it rather than adding a new tool for one deal.
| Tool class | Example (documented behavior) | Best used for | Evidence it can export |
|---|---|---|---|
| Profiler | fg-data-profiling (the project's README says ydata-profiling is now fg-data-profiling): one-line exploratory analysis with type inference, missing-data analysis, duplicates, correlations and automatic alerts | A first look at an extract and a baseline description of every column | HTML report for people and JSON output for automated comparison |
| Assertion framework | Great Expectations (GX Core): Expectations as unit tests for data | Writing explicit rules for the extract: ranges, allowed values, null limits, row counts | Validation results and generated documentation of each run |
| Data contract checker | Soda Core: YAML contracts with missing, invalid, schema and row-count checks | Stating the delivery schema and thresholds as a contract the buyer can read | Pass or fail results per check against the contract |
| Transformation tests | dbt data tests: unique, not_null, accepted_values and relationships, plus source freshness thresholds | Teams that already build the extract in dbt models | Test results with failing-row queries and freshness status |
| Large-scale constraint checks | Deequ on Apache Spark: completeness and uniqueness constraints, profiling, constraint suggestion, anomaly detection | Extracts too large for a single machine | Computed metrics per constraint, stored over runs |
| Monitoring and observability | Scheduled checks on freshness, volume, schema and distributions, run by open source libraries or commercial platforms | Recurring deliveries where the buyer expects consistent refreshes | Alert history and metric trends for each delivered table |
- 01Your work records
- 02Personal details removed
- 03You approve the release
- 04Licensed to AI developers
Data quality tools vs a data licensing review: what can be automated?#
Six quality dimensions can be checked by software with no human judgment beyond setting thresholds. Run them first, because a buyer will find these problems in minutes and they say nothing good about the rest of the package.
The right-hand column matters as much as the check. Each automated pass is necessary evidence, but each leaves a question only the buyer's use case can settle.
| Dimension | What the tool tests | Example check | What a pass does not tell the buyer |
|---|---|---|---|
| Completeness | Nulls, blanks and placeholder values per column | dbt not_null on ticket_id; Soda missing check with a threshold on resolution_code | Why values are missing, and whether the missing rows are the ones the buyer's task needs |
| Uniqueness | Duplicate keys and duplicate rows | dbt unique on the primary key; Deequ uniqueness constraint | Whether near-duplicates, templated text or merged accounts inflate the record count |
| Validity and schema | Types, formats, allowed values, column presence | Soda schema and invalid checks; GX allowed-value Expectation on status | Whether a valid code means the same thing across years and teams |
| Freshness | Age of the newest record against a threshold | dbt source freshness with warn_after and error_after | Whether the history is long and stable enough for training |
| Distribution drift | Shifts in metrics or value mixes between periods or runs | Deequ anomaly detection on stored metrics; profile-to-profile comparison | Whether the shift is a real business change, a system migration or a sampling artifact |
| Referential integrity | Child records point to parents that exist | dbt relationships test from messages to tickets | Whether the joined records form a complete, coherent example for the task |
Why can't a passing test suite answer a buyer's questions?#
Because a test suite checks the data against your rules, while a buyer checks it against their purpose and their legal exposure. A table can pass every test and still be unusable: licensed without the right to train, skewed toward one region, labeled by an auto-close macro, or carrying identifiers in free text.
Standards already make this split. ISO/IEC 25012 defines a data quality model of fifteen characteristics viewed from two angles, inherent and system dependent; both describe the data and its systems, not whether a particular buyer may use it for a particular model. Fitness for a task and permission to use are separate questions.
Keep four ideas apart when you prepare. Access is who can read the system today. Privacy permission is what notices, consents and laws allow for personal information. Ownership is who holds the records. Contractual licensing rights are what customer, vendor and employee agreements let you grant to a third party. De-identification can lower privacy risk, but on its own it does not create a right to license the data.
What does a buyer's acceptance review test that tools cannot?#
A buyer's review adds five judgments that depend on context outside the table: rights and consent, representativeness for the task, label or outcome reliability, residual re-identification risk, and whether the delivered sample replays against the manifest you described. Each needs a person, a document or a decision, not just a query.
| Review question | Why a tool cannot settle it | What the buyer examines | What you prepare |
|---|---|---|---|
| Rights and consent: may this data be licensed for model training? | Rights live in contracts, notices and policies, not in columns | Customer terms, privacy notices, vendor agreements, any restrictions on secondary use | A rights inventory per source system and field group, reviewed by counsel |
| Representativeness: does the data cover the cases the model will see? | Coverage is relative to the buyer's task, which the tool does not know | Slice counts by period, region, product, channel, customer segment and outcome | A coverage matrix with counts per slice and known gaps stated plainly |
| Label and outcome reliability: do outcome fields mean what they say? | A valid code can still be wrong, defaulted or set by automation | Hand-audited samples comparing outcome fields with the underlying record | How each outcome field is set, by whom, and a small audited sample with error notes |
| De-identification residual risk: can people be re-identified? | Pattern scanners miss context; risk depends on what else is known | Free text, rare combinations, dates and locations, linkage to public data | The method used, what was removed or transformed, and the residual risk assessment |
| Sample replay: does the delivery match the manifest? | Replay compares claims to data, so it needs the claims written down first | Random record IDs drawn from the manifest, counts, hashes and field values | A frozen manifest with row counts, date ranges, field list, hashes and the extraction query |
Why do labels and de-identification need human review?#
Label errors pass validity checks because a wrong label is usually a valid value. Northcutt, Athalye and Mueller estimated an average of at least 3.3 percent label errors across the test sets of 10 widely used machine learning benchmarks, with at least 6 percent in the ImageNet validation set, and showed that model rankings can change when test labels are corrected or the share of mislabeled examples rises. Curated benchmarks have that problem; operational outcome fields set by agents, macros and integrations need at least the same scrutiny.
De-identification is a risk judgment, not a pass or fail flag. HHS guidance for the HIPAA Privacy Rule describes two methods, Expert Determination and Safe Harbor, and expert determination requires a person with appropriate statistical and scientific expertise to conclude that the risk of re-identification is very small. NIST SP 800-188, published in September 2023, covers both traditional de-identification and formal privacy methods such as differential privacy for limiting disclosure risk while keeping data useful.
A regex or entity scanner is a useful first pass on free text, and its hit counts belong in your evidence. Treat it as input to the review, not the review itself.
How should you sequence data quality checks before submitting a dataset?#
Run the automated checks on a frozen extract, export their results as evidence, and only then start the human review, so that reviewers spend their time on rights, coverage, labels and privacy rather than on duplicates. The sequence below is the asset of this guide: use it as a pre-submission runbook.
| Step | Run or do | Evidence to export | Owner |
|---|---|---|---|
| 1. Freeze the extract | Snapshot the tables, record the extraction query, date range and row counts | Manifest draft with counts, field list and file hashes | Data engineer |
| 2. Profile every table | Run a profiler on the frozen extract | Profile report in HTML for reviewers and JSON for comparison | Data engineer |
| 3. Assert the basics | Not-null, unique keys, allowed values, schema and row-count checks | Validation results with failing-row counts and examples | Data engineer |
| 4. Check joins | Relationship tests between parent and child tables | Orphan counts per relationship | Data engineer |
| 5. Check time | Freshness of the newest records and volume by month across the full history | Monthly counts chart data and any gaps or migrations explained | Data engineer with system owner |
| 6. Compare periods | Profile or metric comparison between early and late periods | List of drifted fields with the business or system reason | System owner |
| 7. Map rights | Rights inventory by source and field group | Inventory with agreement references and open questions | Counsel and data owner |
| 8. Audit outcomes and coverage | Hand-check a random sample of outcome fields; build a slice coverage matrix | Audit sheet with error notes; coverage counts per slice | Domain expert |
| 9. Review de-identification | Scan free text, assess quasi-identifiers, document the method | Method description and residual risk notes | Privacy lead or qualified expert |
| 10. Replay a sample | Draw random IDs from the manifest and confirm values, counts and hashes reproduce | Replay log with matches and any differences | Someone who did not build the extract |
What should the evidence pack contain?#
The evidence pack should let a buyer reproduce your automated results and see exactly where human judgment was applied. Keep it short and machine-readable where possible.
- The manifest: tables, fields, row counts, date ranges, hashes and the extraction query.
- Tool outputs as files: profile JSON, validation results, test results with failing-row counts, and freshness status, each stamped with the run date and tool version.
- Exceptions explained: every failing or warning check either fixed, excluded with a reason, or accepted with a reason.
- A datasheet-style description: Datasheets for Datasets proposes documenting motivation, composition, collection process, preprocessing and labeling, uses, distribution and maintenance, which maps closely to what buyers ask.
- The human review records: rights inventory, coverage matrix, outcome audit sheet, de-identification method and the replay log.
- A list of known limitations, stated before the buyer finds them.
How does this apply when you license data through SourceX?#
SourceX runs sourcing and qualification, rights review, contracting, data preparation, delivery and payment for businesses licensing operational data to AI developers. Running your own quality tools first still helps: it surfaces fixable problems early and shortens questions later.
The first step is a short company assessment, and no data uploads are needed to see if a company qualifies. Your company approves exactly what is released, once per deal, and sees what would leave before anything is shared. Under the standing offer terms, de-identification is performed by SourceX directly or through its processing partner, and delivery is 30 calendar days after signing unless a deal agrees otherwise.
Frequently asked questions
What are data quality tools?
Data quality tools are software that profile, validate or monitor data against rules: completeness, uniqueness, validity, freshness, drift and referential integrity. Profilers describe a table, assertion frameworks test explicit rules, and monitoring tools rerun checks on a schedule. They do not judge rights, task fit or privacy risk.
Are open source data quality tools enough for a data licensing review?
They are enough for the automated half of the work. Projects such as Great Expectations, Soda Core, dbt tests, Deequ and fg-data-profiling (formerly ydata-profiling) can check and document structural quality. A licensing review still needs people to confirm rights, coverage for the buyer's task, outcome label accuracy and residual re-identification risk.
What are the main dimensions of data quality?
Common frameworks list completeness, uniqueness, validity, consistency, timeliness and accuracy, with names varying by author. ISO/IEC 25012 defines fifteen characteristics viewed as inherent or system dependent. For licensing, add fitness for the buyer's task and the right to use the data, which no dimension list covers on its own.
Can a data quality tool check that data is de-identified?
A tool can scan for patterns such as emails, phone numbers and names and report hit counts, which is useful evidence. It cannot settle residual risk, which depends on rare combinations, free-text context and what outside data could be linked. That judgment needs a documented method and, for health data under HIPAA, Safe Harbor or an expert determination.
Will a buyer rerun my data quality tests?
Often, yes, and on their own copy. That is why you should export results with run dates, tool versions and the exact extraction query, so their numbers can match yours. Mismatches between your manifest and their counts are among the fastest ways to lose a buyer's trust.
Sources
- Great Expectations (GX Core) describes Expectations as expressive and extensible unit tests for your data and can automatically generate documentation for each set of validation results. The repository includes Expectations for value ranges, allowed value sets, non-null proportions, uniqueness and table row counts. Source
- Soda Core describes itself as a data quality and data contract verification engine; contracts are defined in YAML, and its example contract shows missing, invalid, schema and row-count checks. Source
- dbt data tests are SQL queries that look for failing rows and pass when none are returned; the built-in generic tests are unique, not_null, accepted_values and relationships. Source
- dbt freshness checks compare the most recent data with warn_after and error_after thresholds, using loaded_at_field to find the latest load time. Source
- Deequ is a library built on Apache Spark for defining unit tests for data, with completeness and uniqueness checks, data profiling, constraint suggestion and anomaly detection on quality metrics over time. Source
- The README says ydata-profiling is now fg-data-profiling, and describes one-line exploratory data analysis with type inference, missing data and duplicate-row analysis, correlations and automatic alerts, exportable as HTML and JSON. Source
- ISO/IEC 25012 defines a general data quality model with fifteen characteristics considered from two points of view, inherent and system dependent. Source
- Northcutt, Athalye and Mueller (NeurIPS 2021 Datasets and Benchmarks) estimate an average of at least 3.3 percent label errors across the test sets of 10 widely used datasets, with at least 6 percent in the ImageNet validation set, and show that model rankings can change when labels are corrected or the share of mislabeled examples rises. Source
- HHS guidance describes two HIPAA de-identification methods, Expert Determination and Safe Harbor; a person with appropriate statistical and scientific expertise must determine that the risk of re-identification is very small. Source
- NIST SP 800-188, De-Identifying Government Datasets: Techniques and Governance, was published as final in September 2023 and covers traditional de-identification and formal privacy methods such as differential privacy. Source
- Datasheets for Datasets proposes that each dataset be documented with its motivation, composition, collection process, preprocessing and labeling, uses, distribution and maintenance. Source
Related resources
See if your company qualifies
A short company assessment. No data uploads are needed.