How Much Text Do You Need to Adapt an LLM to a New Language?

What decides the token budget for language adaptation

The token budget is set by four variables: baseline competence, tokenizer fit, model size, and the target you measure against. A model that already produces fluent but error-prone text in the language needs far less than one that treats the script as byte fallback. Model size matters because larger models tend to need fewer tokens per unit of improvement on a new distribution, but they also cost more per token to train.

Treat the following as the inputs to every sizing estimate:

• Baseline perplexity and task scores in the target language on a held-out set you control, not on a public benchmark that may already sit in the pretraining data.
• Tokenizer fertility, the average tokens per word. Fertility is the standard tokenization-efficiency metric, and it is high for many non-Latin scripts [4].
• Adaptation method: full continued pretraining, continued pretraining with vocabulary extension, or parameter-efficient methods such as LoRA, which change the volume that is useful.
• Retention target: how much English or source-language regression you will accept on your existing eval suite.

Research on adapting pretrained models to new languages shows that combining tokenizer changes with continued training is an efficient route, rather than training a language-specific model from scratch [1]. That finding is why the sizing question is a continued-pretraining question, which the general guide to how much data to fine-tune an LLM does not cover.

How tokenizer fertility changes the corpus you must buy

Fertility converts a word count from a supplier into a token count for your model, so measure it before you compare quotes. A corpus quoted at 2 billion words is 2.6 billion tokens at a fertility of 1.3 but 6 billion tokens at a fertility of 3.0. The higher number is not more information; it is the same text costing more compute per epoch.

Vocabulary extension lowers fertility and context-window waste, but new embeddings start untrained. Studies of vocabulary expansion for low-resource scripts report that it is not a free win: the new token embeddings need enough target-language text to learn useful representations, which pushes the corpus requirement up [4]. Tokenizer and corpus decisions belong together, so the tokenizer fertility and coverage guide should be read alongside this page.

A practical rule: run your current tokenizer and any candidate extended tokenizer over a 10,000-document sample from each supplier, record fertility per source, and size in your model's tokens, never in the supplier's words or bytes. The token count estimation guide covers sampling methods.

Replay data and catastrophic forgetting

Replay is the main defense against catastrophic forgetting: you keep a share of source-language data in every continued-pretraining batch. Ibrahim et al. show that learning-rate re-warming and re-decaying combined with replay of earlier data lets continual pretraining approach the results of retraining on the combined data, and they test a language shift into German as one of their stronger distribution shifts [2]. Their exact ratios were tuned to their setup, so treat any replay percentage as a hypothesis to benchmark on your own model.

Replay changes the procurement math in two ways. First, total training tokens equal target tokens plus replay tokens, so a 20% replay share on a 10-billion-token target run adds 2.5 billion tokens of source-language text. Second, the replay data must be close to the base model's original mix; if you lack access to it, you need a licensed proxy with comparable domains, which is a separate purchase decision.

Measure forgetting directly. Hold out source-language evals (reasoning, code, instruction following) and run them at each checkpoint; if regressions exceed your tolerance, raise replay before buying more target-language text.

Two-stage adaptation: continued pretraining, then SFT

Many language-adaptation pipelines run continued pretraining on raw target-language text first, then supervised fine-tuning on target-language instructions [4][5]. The two stages need very different data. Stage one needs volume and breadth of native text; stage two needs a far smaller set of high-quality instruction-response pairs written or verified by fluent speakers.

Buying for both stages at once is a common error. Teams overspend on translated instruction data before confirming the base model can generate the language fluently, or they skip native instructions and get a model that understands the language but answers in English. The native vs translated text guide explains why machine-translated text underperforms native text in stage one.

A sizing worksheet for a language-adaptation run

Fill this worksheet before you request quotes; every number in it is a planning hypothesis you revise after a pilot run.

Illustrative example: invented to show structure; it does not describe an available dataset.

The last two rows are where most budgets break. Language-identification errors, boilerplate and duplication remove a large fraction of raw text, so the volume you license must exceed the clean volume you need.

Pilot before committing to the full run. Train at two or three corpus sizes, plot target-language loss against tokens, and buy only when the curve shows where returns flatten for your model.

Where target-language supply runs out

Open supply is bounded, and the bound arrives quickly outside the largest languages. The German Commons, an openly licensed German corpus, reached about 154 billion tokens, which gives a sense of the ceiling for openly licensed text in a major European language [3]. For mid- and low-resource languages, open supply is often orders of magnitude smaller, and crawl-derived text dominates.

Crawl-derived corpora carry three predictable failure modes:

• Language misidentification. Short documents and closely related languages are mislabeled; tools such as GlotLID, a fastText model covering 1,665 languages, were built to reduce this in low-resource settings [7].
• Duplication. Near-duplicates inflate token counts; Lee et al. found one sentence repeated over 60,000 times in C4 [8]. Dedupe before you count.
• Quality pipelines. Building a usable corpus from Common Crawl snapshots requires language-specific filtering, not English heuristics reused [6].

When open supply falls short, the gap is filled with licensed native text: support and sales histories, internal documents, and other operational records written by native speakers in the course of work. The low-resource language text corpora guide and non-English text corpus licensing guide cover acquisition.

Rights and provenance checks before you scale volume

Volume bought without documented rights becomes unusable at training time. The Data Provenance Initiative audit found license omission above 70% and license error rates above 50% on popular dataset hosting sites [9], and low-resource corpus builders report licensing as a central obstacle [10]. Ask every supplier for:

• The rights holder and the chain of consent for each source.
• A license that names pretraining and continued pretraining as permitted uses; see pre-training data license rights.
• Language-ID and dedup reports with the tool and threshold used.
• Per-source token counts measured with your tokenizer.
• Personal-data handling: what was removed or replaced, and how it was checked.

Sourcing target-language text from operational records

SourceX sources operational datasets from US companies on request, including support and sales histories, engineering records, documents, and finance and legal workflows. Nothing is held in stock and a request does not guarantee a match. Every dataset is rights-reviewed for ownership and consents, personal details are removed or replaced before delivery, and each release is approved by the supplying company. For language adaptation, that can suit teams that need native business text in a language rather than a general-web corpus. Buyers can describe the corpus they need on the SourceX buyers page.

For broader volume planning across a purchase, see how much training data to buy, the text data hub, and SourceX's answer to how many records AI labs want. For terminology, see the pretraining glossary entry.

Get target-language text for your adaptation run

SourceX looks for US businesses that hold the data you describe, assesses data and licensing permissions, and delivers under a license that defines records, uses, term and delivery. Nothing is contracted until a supplier agrees, and terms are set per deal. Describe the target-language text you need.