Can you use public benchmarks commercially? Checking eval dataset licenses

This page is general information, not legal advice. Confirm requirements with counsel for your jurisdiction and use case.

What counts as commercial use when you only evaluate

Internal evaluation inside a for-profit company can count as commercial use, even if no benchmark item ever ships to a customer. Non-commercial licenses such as CC BY-NC 4.0 restrict use "primarily intended for or directed towards commercial advantage or monetary compensation," and a model bake-off that decides which vendor you pay, or a regression suite gating a paid release, is hard to describe as anything else. Some dataset authors state that research-only use is fine and say nothing about evaluation; others publish a dataset under NC terms precisely because their upstream sources forced it [1].

There is no settled rule that "evaluation is not use." Running items through a model copies them, scoring stores them, and many teams keep model outputs on benchmark prompts in logs and dashboards. Treat the question as one for legal review, and frame it with facts: who runs the eval, on whose infrastructure, whether results inform a product or a purchase, and whether anything derived from the items leaves the company.

For the broader question of training on non-commercial data, see CC BY-NC and other non-commercial datasets: can a company train on them?. This page stays on the evaluation case.

Why the Hugging Face license field is not enough

The license tag on a Hugging Face dataset is self-reported metadata, not a verified grant. On the Hub, the dataset card is the repository's README.md, and the license: key in its YAML header drives search filters such as "license:mit" [4]. The Hub does not verify that value against the original source, so treat it as a claim to check.

Large audits show how often it is wrong. The Data Provenance Initiative traced more than 1,800 text datasets and reported license omission rates above 70% and error rates above 50% on popular hosting sites [5]. Common failure modes include a permissive tag on a repackaged copy of a non-commercial original, license: other with no text, and a code license (MIT, Apache-2.0) on the loader script that readers mistake for the data license.

A practical rule: the license that binds you is the one granted by whoever held the rights to the content, not the one typed by whoever uploaded the mirror. Good documentation, in the style of Data Cards, names upstream sources, collection and annotation methods and intended use [7]; when a card lacks those fields, assume the audit is still open.

How aggregated benchmarks inherit upstream licenses

A benchmark built from other datasets carries every upstream restriction with it, so a suite is only as permissive as its most restrictive component you actually use. RedBench, for example, aggregates 37 source datasets into 29,362 samples and records the license type of each source [2]. That per-source record is what lets a buyer drop restricted components rather than reject the whole suite.

The Galtea red-teaming set shows the opposite outcome: its maintainers released it under CC BY-NC 4.0 because every prompt came from non-commercially licensed upstream datasets [1]. Retrieval suites behave the same way. The BEIR paper's appendix lists licenses per component, and while some (ArguAna, Touché-2020) are CC BY 4.0, several others restrict commercial use [6]. Our guide to public retrieval dataset licenses walks through that case in detail.

Derived and filtered subsets add a layer. SWE-bench Verified is a 500-sample, human-verified subset of the original SWE-bench [8]; a subset's terms should be read alongside the parent benchmark's and the repositories its tasks came from. Multi-contributor benchmarks such as LegalBench, built from tasks hand-crafted by legal professionals [9], deserve a per-task look rather than a single verdict.

Eval dataset license audit checklist

A defensible audit records five things per benchmark: the stated license, every upstream source and its license, platform or API terms, personal data exposure, and the intended use you are clearing. Research on public benchmarks makes the same point: they need careful auditing for both quality and licensing before you rely on them [3]. Use a license compatibility matrix to resolve combinations, and the open dataset license audit guide for the provenance trace.

Illustrative example: invented to show structure; it does not describe an available dataset.

Keep the record in the same repository as the eval harness, keyed to the pinned revision, so a version bump forces a re-check.

Uses that change the license analysis

The same benchmark can be acceptable for one use and not another, so clear each use separately. The table below sorts common evaluation uses by how much they expose a company.

Illustrative example: invented to show structure; it does not describe an available dataset.

Publishing results raises its own issues, covered in publishing benchmark results on licensed evaluation data. If you plan to negotiate restricted terms, compare them with evaluation-only data license terms buyers should negotiate.

When to keep, restrict or replace a public benchmark

Keep a public benchmark when every component you use has a clear commercial grant; restrict it to non-commercial research when it does not; replace it when the gap blocks a decision you actually need to make. Replacement is also worth considering when a benchmark is likely contaminated, since a license does not fix a test set your candidate models have already seen. See contamination-resistant evaluation design for that side of the decision.

Illustrative example: invented to show structure; it does not describe an available dataset.

Replacement usually means one of three routes: request a commercial grant from the original authors, commission a custom evaluation dataset, or license held-out data drawn from real business records. Each route trades cost against domain fit, and the comparison of private eval sets and public benchmarks covers when held-out data earns its price. For the wider trade-offs between data types, see licensed vs synthetic vs scraped AI training data.

Where licensed evaluation data fits

Licensed evaluation data reduces the upstream-inheritance problem because rights are reviewed at the source and the allowed uses are written into the license. SourceX sources operational datasets from US companies on request, including support and sales histories, engineering records, documents and finance and legal workflows, and manages the licensing process. Every dataset is rights-reviewed for ownership and consents and delivered under a license that defines records, uses, term and delivery. Teams comparing routes can describe the evaluation data they need; a request does not guarantee a match, and nothing is contracted until a supplier agrees.

Personal details such as names, emails, phones and account numbers are removed or replaced before delivery, the method is recorded and a sample is checked, though no method is perfect. That matters for evaluation, where test items are read by many people and often end up in reports. More context on building these sets lives in the evaluation datasets hub.

Sourcing commercially licensed evaluation data

If an audit leaves your product gate resting on a non-commercial benchmark, describe the evaluation data you need rather than the businesses that might hold it. SourceX looks for US businesses that hold the described data, every release is approved by the supplying company, and pricing and allowed uses are agreed in a license per deal. Start a request at sourcex.si/buyers.