Software companies
How to estimate the size of a codebase for a data license
By SourceX Editorial · Updated
Short answer
To estimate the size of a codebase for a data license, measure five things per repository: lines of code by language, commits, merged and declined pull requests with review comments, distinct contributors, and years of active history. Exclude vendored and generated code before counting. Buyers weigh reviewed history far more than raw line counts.
Key takeaways
- Line counts alone mislead, because vendored libraries, generated clients and lockfiles inflate them without adding engineering history.
- Commits, reviewed pull requests and years of activity describe what an AI developer actually learns from: how changes were made and judged.
- Pull requests and review comments are not stored in git, so they must be counted through the hosting platform's API or admin tools.
- A first estimate can use ranges, as long as each number states its method, its exclusions and the date it was measured.
- Running the counts exposes no code; every command reads a local clone and returns totals.
Why does line count alone mislead?#
Line count measures how much text a repository holds, not how much engineering history it contains. A repository can grow quickly from a vendored frontend framework, a generated API client or a checked-in build folder, none of which shows a person deciding anything.
AI developers who license engineering records care mostly about change: the commit that fixed a bug, the review that questioned it, the follow-up that made it work. Two repositories with similar line counts can differ sharply, one holding years of reviewed changes and the other a single bulk import. Count both size and history, and lead with history.
The metric table: what to measure and how#
Each metric below can be produced from a local mirror clone or from the hosting platform without specialist tooling. Run every command against the same clone and record the date, so the numbers agree with each other.
| Metric | How to measure | Notes |
|---|---|---|
| Lines of code by language | cloc --vcs=git in a clean checkout, or scc or tokei | Report code lines apart from comment and blank lines, after exclusions |
| Files and repositories | git ls-files piped to wc -l per repository; repository list from the host | List archived and active repositories separately |
| Commits | git rev-list --count --all --no-merges | Count merge commits separately and remove bot authors |
| Pull requests | Hosting platform API or admin export, grouped by state | Merged and declined both matter; note how many carry review comments |
| Review comments | Comment totals per pull request from the API | Inline comments on code carry more signal than approvals alone |
| Contributors | git shortlog -sne --all, deduplicated with a .mailmap file | Split employees, contractors and bots |
| Years of history | git log --all --reverse --format=%ad --date=short, first and last lines | Also note active months and gaps left by migrations |
| Size on disk | git count-objects -vH on a mirror clone; Git LFS storage separately | Drives the delivery method, not the value |
- 01Your work records
- 02Personal details removed
- 03You approve the release
- 04Licensed to AI developers
What should you exclude before counting?#
Exclusions decide whether the estimate is believable. A buyer who later finds that a large share of the counted lines came from a copied library will discount every other number in the summary, so remove the obvious inflation first and say what you removed.
cloc and similar tools accept excluded directories and file patterns, and many teams already mark generated and vendored paths in .gitattributes. Keep the exclusion list in the repository summary so the count can be reproduced.
- Vendored dependencies: node_modules, vendor and third_party folders, and copied open-source libraries.
- Generated code: protobuf and gRPC output, ORM models, generated API clients, minified JavaScript and build output.
- Lockfiles and manifests that list dependencies rather than describe logic.
- Test snapshots and recorded fixtures, which are counted separately because they need privacy review.
- Data files such as CSV, SQL dumps and large JSON documents.
- Squashed imports: a single commit that brought in an older codebase, which adds lines without history.
How do you count pull requests and review comments?#
Pull requests and review comments live in the hosting platform, so git commands cannot count them. GitHub and Bitbucket call them pull requests and GitLab calls them merge requests; each exposes them through an API that returns state, comments and reviewers. Plan for rate limits: GitHub's REST API allows 5,000 requests per hour with a personal access token, so counting comments pull request by pull request across a large organization can take hours and should run as a resumable script.
Count by state first: merged, declined or closed, and open. Then count pull requests with at least one human review comment, and separate bot comments from linters, CI and dependency tools. That last split matters, because a repository full of automated dependency updates can look busy while holding little human judgment.
Add one linkage count while you are there: pull requests whose title, branch name or commits reference an issue key from Jira or Linear. Linked pull requests connect a reported problem to the change that fixed it and the review that approved it, which is the chain that coding-assistant developers find hardest to source elsewhere. A rough figure is enough, provided you note how keys were matched.
How should you present the estimate?#
Present the estimate as one row per repository with ranges where exact figures are not yet known. At the fit-check stage, a buyer or intermediary needs to understand shape and depth, not audit a precise total, and an honest range with a stated method is more useful than a precise number nobody can reproduce.
Keep the working files behind each row: the cloc output, the exclusion list, the API query used for pull requests and the .mailmap file. When scope is agreed later, the same files let the team rerun the counts on the delivery copy and explain exactly what changed between the estimate and the final package.
| Field | What to enter |
|---|---|
| Repository or alias | A name or code name; aliases are fine at this stage |
| Primary languages | Top languages by code lines after exclusions |
| Code lines | Total after exclusions, with the tool used |
| Commits | Non-merge commits excluding bots, and the first and last commit dates |
| Pull requests | Merged and declined counts, and how many have human review comments |
| Contributors | Employees, contractors and bots, counted separately |
| Known carve-outs | Core IP, customer-owned code or paths with suspected customer data |
What mistakes make a codebase estimate unreliable?#
The most common mistake is counting from a developer's working clone. A working clone mixes in local branches that were never pushed, while a fresh mirror clone holds every server-side ref, and on some hosts that includes pull request refs with commits that were never merged. Count from a fresh mirror and say which refs were included.
Double counting is the second. Forks and repositories copied with their full history repeat the same commits, so deduplicate by commit hash when summing across repositories. Repositories split out of a monorepo usually carry rewritten copies of old commits with new hashes, so compare their early history with the monorepo before adding them together.
Automation is the third. Dependency update bots, release tooling and code formatters can produce a large share of commits and pull requests in a busy repository. Filter authors by bot account names and email patterns before reporting, and give the bot totals separately so nobody suspects they were hidden.
Illustrative: a property management software vendor counts its repositories#
Illustrative: a fictional property management software vendor runs a large monorepo for its current product plus several older repositories for a retired desktop client. The first cloc run is dominated by a vendored frontend framework and a generated payments API client.
After exclusions, the monorepo's line count drops noticeably, but its commit and review history is long and dense. The desktop client's repository turns out to be a single squashed import from an older version control system, so its history is shallow. The CTO leads the summary with the monorepo's reviewed pull requests, lists the desktop client as a snapshot-only candidate, and flags one contractor-heavy repository for an IP assignment check.
How SourceX uses a codebase estimate#
SourceX uses the estimate in the Supply step of the SourceX five-step transaction, where only metadata is collected. Counts, languages, date ranges and carve-outs are enough to judge fit; no repository is cloned or shared at this stage.
In the SourceX Enterprise Data Value Framework, scale is one driver among several, alongside human-generated signal, recency and rights, while preparation cost reduces net value. A smaller repository with dense review history and clear ownership can therefore matter more than a larger one with thin history, which is why the estimate should report both.
Frequently asked questions
Do we need exact numbers for a first conversation?
No. Ranges with a stated method are enough to judge fit. Exact counts become useful later, when scope is agreed and preparation effort is planned, and by then the delivery copy will have its own exclusions that change the totals anyway.
Should we count code for products we no longer sell?
Yes, but list it separately. Retired products can be licensable and are often simpler to approve because nothing about them is competitive today. Their history may be shallower if they moved between version control systems, so count commits and pull requests rather than assuming depth from age.
How do we count a monorepo against many small repositories?
Count a monorepo by top-level product or service folder, using path filters in cloc and git log, so it can be compared with separate repositories. Buyers think in terms of products and workflows, and a single monorepo total hides which parts carry the history.
Should test code count toward the estimate?
Yes, reported as its own line. Test code shows how engineers specify expected behavior, and tests written alongside a fix are useful evidence of how the fix was verified. Keep recorded fixtures and snapshots out of that figure, because they are data rather than code and need a privacy review first.
Does running these commands expose any code?
No. Every command reads a local clone and prints totals, and the API calls for pull requests return counts and metadata to your own machine. Nothing needs to leave the company to produce the estimate.
Do contractor commits change the estimate?
They do not change the counts, but they can change the licensable scope. Code written by contractors is owned by the company only if their agreements assigned the IP, so flag repositories with heavy contractor activity for a rights check.
Sources
- GitHub's REST API allows 5,000 requests per hour for requests made with a personal access token. Source
Related resources
See if your company qualifies
A short company assessment. No data uploads are needed.