How to measure language share of a dataset: an English-share worksheet

What does the language share of a dataset actually measure?#

The language share of a dataset is the fraction of a stated unit, such as records, tokens or audio minutes, that is in a given language under a stated counting rule, measured over a defined scope. Change any of those three and the number changes, sometimes by many percentage points, so a share quoted without them is not a measurement anyone can check.

Scope comes first. Write down which systems, which date range and which filters define the offer: for example, Zendesk tickets from the North America brand, 2021 to 2025, closed status, excluding internal notes. A company-wide language mix is a different number from the share in the records you would actually offer, and only the second one matters to a buyer.

Language share is also not the same as the language your team works in. An English-speaking support team can still hold Spanish customer messages, Portuguese attachments and auto-translated replies, and the records carry whatever language the customer wrote in. Measure the records, not the org chart.

This page is about producing a defensible number as the data holder. If you are on the receiving side and need to check a supplier's language labels, see the guide to language identification QA for multilingual datasets linked below.

Which unit should you count: records, tokens, characters or audio minutes?#

Count in the unit the buyer will use the data in: tokens or characters for text training, audio duration for speech, and records only when each record is the unit of use, such as one ticket as one example. Report at least two units when they differ, because a record count can hide a large volume of non-English text inside a few long records.

Tokens depend on the tokenizer, so name it. OpenAI's tiktoken documentation notes that a token corresponds to about 4 bytes of text on average in practice, and that ratio shifts with language and script, which is why a token share computed with one tokenizer will not match another. Characters are more neutral across tokenizers; count user-perceived characters rather than bytes when scripts differ.

How should mixed-language records be counted?#

Pick one counting rule for mixed-language records before you measure, write it down, and apply it to every record. The most transparent option is to split each record into segments, count each segment in its own language, and report mixed records as a separate bucket so a reader can see how much depends on the rule.

Mixed records are normal in operational data. A support ticket may hold a Spanish customer message and an English agent reply; an email thread may quote a Portuguese message below an English answer; a CRM note may switch languages mid-sentence. Strip what is not authored content before counting: signatures, legal footers, templated macros, system notifications and quoted reply history, otherwise a repeated English footer inflates the English share.

Some content has no language at all, such as order numbers, URLs, stack traces or attachment-only records. Keep it out of the language numerator and report it separately. Use separate buckets for no linguistic content and for undetermined language, so your report is easier to audit.

Which language identification tools work, and how often are they wrong?#

Use an established language identification (LID) model, pin its version, set a confidence threshold, and check its output on a human-reviewed sample because short and noisy text is where these tools fail most. The tool gives you a fast first pass; the human-reviewed sample is what makes the number defensible.

Short text is the main weakness. The Lingua project's own German report shows 74.2% accuracy on single words averaging 9 characters, 93.9% on word pairs and 99.7% on sentences averaging 111 characters. GlotLID's maintainers advise against using their model on very short sentences and note that the right confidence threshold differs by language. Operational data is full of short fragments, so set a minimum length below which a segment goes to the undetermined bucket.

Errors also run in a direction that matters for an English share. GlotLID's documentation says its false positive rate for high-resource languages is much higher than for low-resource ones, which means text in other languages can be mislabeled as a high-resource language such as English. Verify any model's labels on a human-reviewed sample before relying on them.

How big a sample do you need to estimate the English share?#

For a record-level share, a simple random sample of about 385 records gives a 95% confidence interval of roughly plus or minus 5 percentage points in the worst case, and about 1,068 records gives roughly plus or minus 3 points. Those figures come from the standard normal-approximation interval for a proportion with the share assumed to be 50%, the most conservative guess.

The interval is the sample proportion plus or minus a critical z-value (1.96 at 95%) times the square root of p(1 minus p) over N. It is an approximation that works when N is large enough. Assuming p equals 0.5 gives the largest required sample, so it is the safe planning choice when you have no prior estimate. For small samples, or shares near 0% or 100%, use exact binomial limits instead.

Three practical adjustments matter more than the formula. Stratify the sample by source system, channel, region or year, because language mix often differs sharply between them, and weight results back to the population. If the unit is tokens or characters, sample records but compute the share as English characters over all characters in the sample, and estimate its uncertainty by resampling records, since a few long records can dominate. And record the random seed and sampling frame so someone else can draw the same sample.

• 1Plus or minus 5 points at 95% confidence: about 385 records, worst case.
• 2Plus or minus 3 points at 95% confidence: about 1,068 records, worst case.
• 3Plus or minus 2 points at 95% confidence: about 2,401 records, worst case.
• 4Each stratum you want to report on its own needs its own adequate sample.

Worked example: the English share of a support-ticket export#

Illustrative example, invented to show the method; it does not describe a real company or dataset. A software company plans to offer 180,000 closed Zendesk tickets from 2021 to 2025. It strips signatures, macros and quoted history, draws a random sample of 400 tickets stratified by channel, runs two LID models at the comment level, and has a reviewer check every ticket where the models disagree or confidence is low.

In the sample, 300 tickets are entirely English, 40 are entirely Spanish, 44 mix a Spanish customer message with English agent replies, and 16 contain no linguistic content. Of the 44 mixed tickets, 20 are majority English by characters. Across the 384 tickets with language content, 612,000 of 760,000 characters are English.

The same sample therefore supports four different headlines: 75% English under the strict record rule (300 of 400), 80% under the majority rule (320 of 400), 86% under the any-English rule (344 of 400), and about 81% by characters. Excluding the 16 no-language tickets from the denominator moves each record figure up again. With 400 records, the strict-rule estimate carries a 95% interval of roughly plus or minus 4 points.

The honest report states the scope, unit, rule, method and interval for at least two of these, shows the mixed and no-language buckets, and labels the result a measured estimate. Picking the 86% figure without its rule would overstate what a buyer receives.

How should you report an estimate versus a verified figure?#

Label every language-share figure with its status: unknown, supplier estimate, measured estimate or verified. Unknown is not zero, an estimate is not a verified number, and a band such as most or nearly all is not a percentage. Keep the interval with the point estimate and do not round up to a convenient figure.

Metadata alone is not a measurement. The Hugging Face dataset card format, for example, lists a dataset's languages as ISO codes in its metadata, which says which languages appear but not how much of each. Add an explicit share statement with unit, rule, method and date. If a form asks for a range, choose the range that contains your whole interval, or say that the interval straddles two ranges.

Re-measure when the scope changes. Removing a brand, adding a year or applying de-identification can shift the language mix, and the figure that matters is the one for the records actually delivered.

Language-share measurement worksheet#

Fill in one worksheet per offered scope and keep it with your data profile. Every row should be answerable by someone who was not in the room when the measurement was made.

How this applies when licensing data through SourceX#

A measured language share belongs in the description of your data, next to volume, date range and record types, so the first conversation is about the right records. SourceX works with U.S., Canadian and EU companies that had 50 or more full-time employees at their peak, and the first step is a short company assessment that needs no data uploads.

AI developers who request data through SourceX state the language they need alongside scope, volume, format, timing and licensing needs, so state your language share in terms a request can be matched against. Before anything is shared, the company sees and approves exactly what would be released, once per deal; a language figure measured on that exact scope is the one worth stating.

Do not translate non-English records to raise the share. Translated text is a different kind of data from natively written text, and it changes what a buyer would be licensing; if only an English subset is offered, define that subset and measure it on its own.

SourceX can consider companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when the data they offer is at least 90% English, confirmed on the actual dataset; every dataset still goes through rights, privacy, quality and buyer-fit review.