English-only subset of multilingual data: how to scope and document it

What is an English-only subset of multilingual data?#

An English-only subset is a separately defined slice of a multilingual archive that contains only records originally written or spoken in English, selected by a written rule and documented as its own asset. It is not the archive with the other languages translated away, and it is not a promise that every record in the company is English.

Companies with international customers usually hold mixed archives: a help desk with English, Spanish and French tickets, a knowledge base published in several locales, or a CRM where account notes follow the rep's language. When a buyer asks for English text, the practical answer is often to carve out the English portion rather than offer everything.

The subset only works if a stranger can reproduce it. That means four things written down before anything is exported: the boundary rule, how translated and mixed records are treated, where each record came from, and what was left out and why.

Which boundary rule should define the subset?#

Use the most reliable language signal your systems already store, then confirm it with content-level language identification on a sample. No single signal is complete, so a written combination usually beats any one field.

System metadata is the cheapest place to start. Zendesk Help Center articles carry a source_locale and an outdated_locales list, and Salesforce Knowledge marks the primary-language version of each article with IsMasterLanguage. Ticket queues, brands or regional routing groups can also signal language, but they describe where work was routed, not what was written.

Locale tags can carry a region as well as a language. Under BCP 47, en-US and en-GB both mean English, and the region subtag marks regional variation. Decide up front whether the subset is any English or a named variety, and do not read a region subtag as the customer's country.

Why are translated records not original English records?#

A translated record is a new text derived from a source in another language, so it carries different authorship, different language patterns and often different rights. Treat it as its own class, outside the original-English subset unless the buyer explicitly asks for it.

The language difference is measurable. Machine translation research calls the distinct style of translated text translationese: Zhang and Toral found that translated test data inflates human evaluation scores and can change system rankings, and Graham, Haddow and Koehn recommend leaving reverse-created test data out of future test sets. A buyer who asked for natural English is buying exactly the property that translation removes.

The rights difference matters too. U.S. copyright law lists a translation as an example of a derivative work, and the Berne Convention gives authors the exclusive right to make and authorize translations of their works. A translation made by a vendor, a contractor or a machine translation service may come with its own ownership questions and service terms, separate from the original record.

• 1Help center translations: exclude every non-source locale version, and check whether the English version is itself the source or a translation of another language.
• 2Tickets with built-in translation: keep the customer's original text and drop the machine-rendered English, or keep both only as a labeled pair if the buyer wants it.
• 3Human-translated documents: exclude them from the original-English subset and list them as a separate, optional class with their own rights note.
• 4Post-edited machine translation: treat it as translated, not original, even after heavy human editing.

How should mixed-language records be counted?#

Write one rule for records that contain more than one language, apply it the same way to every record, and report how many records it affected. Without a rule, two people will draw different boundaries on the same export.

Mixed records are common in operational data. A support thread can open in Spanish and continue in English, an email can quote a French message below an English reply, and an engineering ticket can be mostly stack traces with one English sentence. Language detectors handle these poorly: the GlotLID documentation warns against using its model on very short sentences and notes that confidence thresholds differ by language.

• 1Whole-record rule: include a record only if every human-written message in it is English. Simple and strict; drops many useful threads.
• 2Segment rule: split threads into messages, keep the English messages, and record the parent ID so context gaps are visible. More yield, more processing.
• 3Majority rule: include a record if most of its human-written text is English, and tag it as mixed. Easy to apply; the buyer must accept it.
• 4Non-language content: set aside code, logs, numbers and signatures before measuring, and tag them as having no linguistic content rather than counting them as English.
• 5Quoted text and attachments: decide whether quoted earlier messages and attached documents are in scope, and apply the same language rule to them.

What provenance does each record in the subset need?#

Every record in the subset should point back to its original record in the source system and say how it was selected. The subset is derived from the archive, and the documentation should show that link.

The W3C PROV ontology gives the vocabulary: a subset is an entity derived from a pre-existing entity, the archive, through a selection activity. You do not need RDF to use the idea. A few columns per record are enough to let a reviewer trace any row to its source and rerun the selection.

Standard dataset metadata can carry the subset-level facts. Croissant lets a dataset declare its content languages with inLanguage and define a file set with include and exclude patterns. Hugging Face dataset cards have fields for language, language_creators, BCP 47 language_details such as en-US, source_datasets, and named configs for subsets.

Do the archive's rights and privacy terms carry over to the subset?#

Selecting a subset narrows what is offered; it does not add rights the company did not have for the full archive. Every rights and privacy question that applies to the archive still applies to each record in the subset.

Keep four questions separate. Access means the company can export the records. Privacy permission means the law and the company's notices allow this use of personal data. Ownership means who holds rights in the content. Contractual licensing rights mean what customer, vendor and platform agreements allow the company to grant a buyer. A record can pass one test and fail another.

Language does not decide which privacy law applies to a record; where the people are and where the company operates usually matter more, and counsel should confirm that for the specific archive. An English ticket from a customer abroad can carry the same obligations as a non-English one, and de-identification alone does not create a right to license the records.

Which quality checks show the boundary held?#

Check both sides of the boundary: sample what you included for records that should be out, and sample what you excluded for English records that should be in. Report the error found on each side, not just the subset size.

• 1Inclusion sample: review a random sample of included records for non-English text, translations and machine-rendered English.
• 2Exclusion sample: review a random sample of excluded records for original English that the rule dropped, especially short replies.
• 3Translation duplicates: in knowledge bases, confirm only one language version per article remains; in Salesforce, the Article Number groups a primary article with its translations, so it is a useful dedupe key.
• 4Stale or swapped sources: flag articles whose source locale was changed or whose translations are marked outdated, and confirm which version is original.
• 5Reconciliation: confirm that included plus excluded counts equal the parent archive count for the date range.
• 6Detector agreement: where metadata and language identification disagree, route the record to human review rather than letting either one decide.

How do you document what is excluded?#

Keep an exclusion log that lists every class of record left out, the rule that removed it, how many records it covered and whether it could be offered separately later. Exclusions are part of the subset definition, not a footnote.

Datasheets for Datasets asks whether a dataset holds every possible instance or is a sample of instances from a larger set. An English-only subset is exactly that kind of sample, and the exclusion log is how you answer the question honestly. The counts below are illustrative only.

Subset scoping template: what should the definition contain?#

Fill in this template before export and give it to the reviewer with the subset. It turns the decisions above into one page that a buyer, counsel and your own team can check against the data.

How this applies when licensing data through SourceX#

SourceX runs sourcing and qualification, rights review, contracting, data preparation, delivery and payment for businesses licensing operational data to AI developers. It works with U.S., Canadian and EU companies that had 50 or more full-time employees at their peak, contractors excluded.

The first step is a short company assessment, and no data uploads are needed to see if a company qualifies. If a buyer's request calls for English text and your archive is multilingual, a written subset definition like the one above makes the scope concrete, and your company approves exactly what is released, once per deal, before anything is shared. De-identification is performed by SourceX directly or through its processing partner.

SourceX can consider companies in the UK, Australia, New Zealand, India, Singapore, South Africa, Latin America, the UAE and Saudi Arabia when the data they offer is at least 90% English, confirmed on the actual dataset; every dataset still goes through rights, privacy, quality and buyer-fit review.