Code-Switched and Romanized Text Data for Multilingual Models

Why web-scale corpora under-supply mixed-language text

Mixed-language text is scarce in most training mixes because corpus pipelines assume one language per document. Language identification is a filtering step in corpus building, and its errors decide which text survives, a problem studied most for low-resource languages [3]. A message that is half Spanish and half English may get one label, a low confidence score, or a language bucket that is then down-sampled; we treat this as a working hypothesis to test on your own pipeline, not a measured rate. Researchers describe this "monolingual assumption" as a core limitation of current models and datasets [6].

The public alternatives are mostly research benchmarks. LinCE combines code-switched corpora for Spanish-English, Nepali-English, Hindi-English and Modern Standard Arabic-Egyptian Arabic across language ID, NER, POS tagging and sentiment tasks [1]. CodeMixBench evaluates LLM code-mixing across 18 languages [7]. These are useful for evaluation, but they are small, often social-media derived, and rarely reflect the register of customer service or field operations. For the broader case for non-web text, see proprietary text data beyond web crawls.

US Spanish-English mixing in business text

For teams serving bilingual US customers, the most valuable source is operational: support chats, SMS threads, call-center agent notes and sales follow-ups where customers and staff switch freely. Typical patterns include intra-sentential switches ("necesito cancelar la subscription"), English business terms inside Spanish clauses, and nonstandard spelling without accents. A widely used public reference, the Bangor Miami corpus, holds transcribed informal conversations recorded in Miami and carries its own access terms, so check them before any training use [5].

What to specify in a request:

• Channel and direction. Customer-to-agent, agent-to-customer, or internal notes; each has a different switch rate and formality.
• Switch density. The share of messages with at least one switch, and whether all-English and all-Spanish messages from the same threads are included for contrast.
• Domain. Billing, logistics, healthcare scheduling or insurance claims vocabulary changes what "domain term in English" looks like.
• Speech or text. If you also need audio, the companion page on code-switched speech data covers transcription conventions.

Romanized South Asian and Arabic chat text

Romanized text is a script problem, not only a language-mixing problem, and it needs its own section of your spec. Hindi, Urdu, Bengali, Tamil and other South Asian languages are widely typed in Latin letters with no standard spelling, so one word can appear in many variants ("nahi", "nahin", "nai"). The Dakshina dataset covers 12 South Asian languages with native-script text, a romanization lexicon and native/Latin parallel sentences, and reports transliteration and language-modeling baselines [2]. It is a strong reference for evaluation, but its sentences come from Wikipedia rather than chat.

Arabic chat alphabet ("Arabizi") adds digits for sounds without Latin equivalents, such as "3" and "7", which tokenizers and normalizers may split or strip. Ask suppliers whether digits inside words were preserved, whether any automatic transliteration back to native script was applied, and whether the original keyboard input is retained. Dialect matters too: the dialect and regional variety guide covers standard varieties, while this page covers the mixing and script layer on top.

Language tags, labels and the labeling method

Span-level language tags are the single most useful annotation for code-switched text, and the labeling method matters as much as the tags. LinCE-style token-level tags (lang1, lang2, mixed, named entity, ambiguous, other) let you compute switch points and filter by density [1]. Ask whether tags came from a classifier, human annotators, or a classifier with human review, and request an agreement score on a sample.

Illustrative example: invented to show structure; it does not describe an available dataset.

{
  "record_id": "msg_000184",
  "channel": "support_chat",
  "speaker_role": "customer",
  "text": "hola, ya pagué el bill pero la app dice [AMOUNT] pending",
  "script": "Latn",
  "romanized": false,
  "spans": [
    {"start": 0, "end": 17, "lang": "es"},
    {"start": 18, "end": 22, "lang": "en"},
    {"start": 23, "end": 39, "lang": "es"},
    {"start": 40, "end": 48, "lang": "placeholder"},
    {"start": 49, "end": 56, "lang": "en"}
  ],
  "tag_method": "classifier+human_review",
  "deid_method": "rule+ner_replace_v2",
  "thread_id": "t_00921"
}

Track documentation in a machine-readable form. Croissant-RAI extends Croissant metadata with responsible-AI fields, including data labeling and life-cycle documentation, which fits tag methods and de-identification notes [9]. For checking language labels at scale, see language identification QA for multilingual datasets.

Tokenizer fertility and vocabulary checks

Measure tokenizer fertility on a supplier sample before you buy, because romanized and mixed text can tokenize very differently from either source language. Fertility, the average tokens per word, varies by language and tokenizer, and vocabulary extension is one way teams reduce it [4]. Run your production tokenizer on 1,000 to 5,000 sampled messages, split by script and switch density, and compare against monolingual baselines from the same channel.

High fertility on romanized Hindi or Arabizi raises training and inference cost and can hurt quality on short messages. The tokenizer fertility and corpus coverage guide explains how to decide whether a corpus justifies vocabulary extension, and token count estimation helps you size the license.

Buyer evaluation checklist

Use this checklist to compare supplier samples for SFT, classification, tokenizer coverage or evaluation sets.

Illustrative example: invented to show structure; it does not describe an available dataset.

If authenticity matters, the guide to verifying human-written text covers checks for machine-generated content, and native vs translated text explains why translated mixing is a poor substitute.

De-identifying mixed-language chats

De-identification is harder in code-switched text because PII detectors are usually tuned per language. A Spanish surname inside an English sentence, or a romanized Hindi name that resembles a common word, can slip past an English-trained recognizer. Presidio, a common open-source option, uses ML models and itself warns there is no guarantee it will find all sensitive information [8].

Ask for per-language recall on a labeled sample and consistent placeholders such as [NAME] and [AMOUNT] that do not break span tags. The chat-log de-identification guide walks through methods. SourceX removes or replaces personal details such as names, emails, phones and account numbers before delivery, records the method and checks a sample, though no method is perfect.

How SourceX sources mixed-language business text

SourceX sources operational datasets from US companies, including support and sales histories, and manages licensing and ongoing purchases. Data is sourced on request rather than held in stock, so a request does not guarantee a match, and every release is approved by the supplying company. Related owner pages cover licensing chat logs and customer support transcripts; you can also start from the text data hub or describe your requirement to SourceX.

Each dataset is rights-reviewed for ownership and consents and delivered under a license defining records, uses, term and delivery. SourceX does not source scraped web content, and what it may do with a deidentified dataset is set out in the signed supplier agreement.

Request code-switched text data

Describe the language pairs, scripts, channels and tags you need, and SourceX looks for US businesses that hold matching data, then assesses data and licensing permissions before anything is agreed. Nothing is contracted until a supplier agrees, and pricing and allowed uses are set in a license per deal. Start a buyer request at SourceX.