Building an Accent-Stratified Evaluation Set for ASR Fairness

Why a general test set hides accent disparities

A single pooled WER hides accent gaps because the speakers who dominate the test set also dominate the average. The authors of the Edinburgh International Accents of English Corpus (EdAcc) report that the best model they evaluated, trained on 680k hours of transcribed data, reached a 19.7% average WER on their conversational accented speech against 2.7% on US English clean read speech, with notable drops on varieties such as Indian and Jamaican English [1]. A model can look production-ready on a LibriSpeech-style benchmark and still fail a large share of real users.

Earlier service audits measured the same pattern on read speech, using a public accent archive with more than 2,700 speakers from 171 countries [2]. That design is useful for a first screen, but read elicitation paragraphs differ from conversational production audio in disfluency, speaking rate, vocabulary and channel. Treat archive results as a lower bound on the disparity you will see in calls, meetings or dictation, and see why read and scripted speech fall short for the mechanics.

Defining strata that survive review

Strata should be defined by a documented taxonomy and a documented labeling method, because "accent" collected three different ways cannot be compared. Purpose-built sets organized by English variety, such as World Englishes evaluation work, show that the grouping itself is a design decision to justify, not a column to inherit [3]. Decide up front which axis you are auditing.

• L1 or country of upbringing: self-reported by the speaker at consent time; stable and explainable, but coarse (for example, "Hindi L1" spans many regional varieties).
• Regional or social dialect: for US-only products, groups such as African American English, Southern US English or Chicano English matter more than L1; see US regional and social dialect speech data.
• Perceived accent: labeled by trained annotators listening to audio; closest to what the model "hears," but it needs inter-annotator agreement reporting.
• Proficiency or years of exposure: a secondary axis that separates L2 learners from fluent speakers who share an L1.

Pick one primary axis and record the others as covariates. Then add the factors that confound accent: microphone and channel (8 kHz telephony versus wideband), signal-to-noise ratio, speaking style, age band and domain vocabulary. If your Indian English stratum is mostly mobile-phone audio and your US stratum is headset audio, you are measuring channel, not accent. The speech metadata fields buyers should require page lists the field names to put in a request.

Sizing: count speakers, not utterances

Statistical power in an accent audit comes from the number of distinct speakers per stratum, because errors cluster within a speaker. Five hundred utterances from six speakers tell you about six people; a single unusual voice can move that stratum's WER by several points. Treat speaker as the sampling unit and compute confidence intervals with a speaker-level (cluster) bootstrap: resample speakers with replacement, recompute corpus WER, and repeat 1,000 or more times.

A practical working rule, which you should validate against your own variance, is to target dozens of speakers per stratum and a fixed minimum of audio per speaker, then check that the confidence interval on each stratum's WER is narrow enough to detect the smallest gap you would act on. Balance gender and age bands within strata where consent allows, or at least report them. For sizing math in general, see eval set size and statistical power, and for allocating examples to small but important groups, see stratified evaluation sets for rare and high-risk cases.

Illustrative example: invented to show structure; it does not describe an available dataset.

The key property is the "Same as reference" column: every stratum is matched on channel and style so the gap you measure is attributable to the speaker group.

Reference transcripts and normalization decide the result

Accent WER gaps are only meaningful if every stratum is transcribed under the same convention and normalized the same way before scoring. Rev researchers argue that datasets vary in transcription style and formality, that this distorts scores, and that standard WER likely overstates the contentful errors of strong systems; multiple references reduce the distortion [4]. The Whisper authors built a text normalizer so harmless spelling and formatting differences would not count as errors, and checked it against an independent normalizer from the FairSpeech project [5].

Three accent-specific failure modes recur. First, dialect forms such as habitual "be" or "ain't" get "corrected" by transcribers into standard grammar, so the reference itself penalizes a model that transcribed what was said. Second, transcribers unfamiliar with a variety mishear it, and their errors land disproportionately in that stratum. Third, code-switched words and local place names get inconsistent spellings. Use transcribers familiar with each variety, double-transcribe a sample per stratum, adjudicate disagreements, and publish your verbatim versus clean transcription standard.

Test-set labels are never perfect: an audit of widely used vision, NLP and audio benchmarks estimated an average test-set label error rate of at least 3.3% [6]. If reference errors are uneven across strata, they create or erase a gap. Report reference-audit error rates per stratum alongside WER.

Metrics and a reporting format that reviewers accept

Report corpus-level WER per stratum, the absolute and relative gap to a declared reference group, and speaker-level confidence intervals. Compute stratum WER as total substitutions, deletions and insertions divided by total reference words, not as the mean of utterance WERs, which overweights short utterances. Then add:

• Worst-group WER and the max-minus-min gap across strata.
• Error-type breakdown (substitution, deletion, insertion) per stratum; high deletion often signals the model dropping speech it cannot decode, and insertions can signal hallucinated text.
• Speaker-level distribution: the share of speakers in each stratum above a usability threshold you define, which shows whether a gap is broad or driven by a few voices.
• Entity or keyword error rate on names, numbers and domain terms, where accent errors usually cause the most harm; see domain vocabulary coverage.

Illustrative example: invented to show structure; it does not describe an available dataset.

{
  "utterance_id": "eval-ae-0412-017",
  "speaker_id": "spk-0412",
  "stratum_primary": "en-AAE-US",
  "l1_self_report": "English",
  "country_of_upbringing": "US",
  "region_self_report": "Georgia",
  "age_band": "35-44",
  "channel": "telephony_8khz_mulaw",
  "snr_db_estimate": 18,
  "speech_style": "conversational_two_party",
  "domain": "billing_support",
  "reference_convention": "verbatim_v2_dialect_preserving",
  "reference_pass": "double_transcribed_adjudicated",
  "consent_scope": "evaluation_only",
  "split": "eval_locked"
}

For audit trails, attach a dataset card that documents sources, collection, annotation method, intended use and known limits [8], and consider expressing it in machine-readable Croissant-RAI so the metadata travels with the files [7]. In the NIST AI RMF, this work sits in the MEASURE function, with stratum definitions and thresholds belonging in MAP and GOVERN [10].

Keeping the set held out and uncontaminated

An accent fairness set only measures generalization if none of its speakers or recordings appear in the training data of the systems you score. Public corpora are the main risk: crowdsourced read-speech collections such as Common Voice are public domain and widely used in training [9], so a vendor's low WER on them may reflect exposure rather than robustness. Check licenses before any use with the open speech corpora commercial license audit.

Enforce speaker disjointness by speaker ID and, where possible, by voice similarity check against your own training pool. Lock the set: version it, restrict access, do not send full audio to vendor APIs that retain inputs, and rotate a fresh slice when a stratum saturates. The held-out data definition and keeping a private eval set private cover access controls and canaries. When comparing ASR vendors, run all candidates on the same locked set under the same normalizer, as in a vendor bake-off on a private eval set.

Sourcing the audio: archives, purpose-built collection or licensed production speech

The right source depends on whether you are screening, certifying a launch or comparing vendors. Public accent archives are fast and cheap for screening but are read speech [2]. Purpose-built collection gives you control over strata and consent but tends to produce scripted or semi-scripted speech. Licensed real-world audio, such as recorded support or sales calls, best matches production conditions but needs speaker-level accent metadata, consent that covers evaluation use, and removal of personal details before delivery.

When writing a request, specify the strata table, minimum distinct speakers per stratum, channel and style matching, metadata fields, the transcription convention and the license scope. For eval sets, ask that the license restrict use to evaluation and prohibit training, and that it state how the set may be shared with ASR vendors under test. Redaction of names, numbers and account details in audio creates silences or tones that can themselves change WER, so agree how redacted spans are marked in references; see how audio redaction affects speech models. For training-side coverage, the companion guide on accented English speech data for ASR applies.

SourceX sources operational datasets, including support and sales histories and new recordings of hands-on work, from US companies on request, rather than holding stock, and a request does not guarantee a match. Every dataset is rights-reviewed for ownership and consents, personal details are removed or replaced before delivery with the method recorded and a sample checked, and delivery runs through private, access-controlled workflows after an executed agreement. Buyers describe the speech they need on the SourceX buyer page.

Regulatory context for testing data

As of October 2026, regulation treats testing data as part of data governance for high-risk systems, not only training data. EU AI Act Article 10 requires high-risk AI systems that are trained with data to be developed on training, validation and testing data sets meeting quality criteria and governance requirements [11]. Reg. (EU) 2026/1744 reportedly moved the high-risk application dates to 2 December 2027 (Annex III) and 2 August 2028 (Annex I), and also amends Article 10 [12]. If your ASR feeds a high-risk use, such as screening in employment, a documented accent-stratified test set is direct evidence for that governance file.

This page is general information, not legal advice. Confirm requirements with counsel for your jurisdiction and use case.

Request accent-stratified speech for ASR evaluation

SourceX looks for US businesses that hold the speech data you describe, rights-reviews it, and delivers it under a license that defines records, uses, term and delivery, with the supplying company approving every release. Nothing is contracted until a supplier agrees. Describe your strata, channels and speaker counts at sourcex.si/buyers.

See also the speech and audio data buyer's guide and evaluation datasets built from real business work.