Start with what Whisper already heard
Fine-tuning doesn't start from zero. It starts from whatever the model learned in pretraining, so the first question isn't "how many hours do I need?" but "how many hours of my kind of speech has this model already heard?"
OpenAI trained the original Whisper on 680,000 hours of audio paired with transcripts. Of those, 117,000 hours covered 96 languages other than English and 125,000 hours were translation data, which leaves roughly 438,000 hours of English. The paper is blunt about what that means for everything else: most of those 96 languages had less than 1,000 hours of training data, and Whisper's speech recognition was still quite poor on many of them.
The newer large-v3 model was trained on 1 million hours of weakly labeled audio and 4 million hours of pseudo-labeled audio, but its model card doesn't break that down by language. The original paper does chart how much training data each language got (Figure 3 and Appendix E), and it's worth five minutes before you plan a budget.
If your language had tens of thousands of hours in pretraining, you're adapting a model that already speaks it. If it had tens of hours, you're closer to teaching it. Those two projects need very different amounts of data.
The 16x rule, and why it matters for your budget
The most useful number in the Whisper paper isn't 680,000. It's the relationship the authors found between how much training data a language had and how well Whisper recognized it.
"WER halves for every 16× increase in training data."
Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision, 2022Across languages, the log of a language's training hours and the log of its word error rate (WER) on the FLEURS benchmark lined up closely, with a squared correlation of 0.83. Put plainly, each halving of the error rate took 16 times the training data.
Each halving of the error rate needs 16 times the data
Word error rate relative to a starting point, as a language's training hours grow. Based on the trend reported in the Whisper paper.
That trend comes from pretraining across many languages, not from one fine-tuning run, so don't treat it as a promise. But expect the same shape when you fine-tune: the first hours of well-matched data move the error rate a lot, and later hours move it less. Two things follow from that:
- Big jumps in quality need big jumps in data. Going from 50 to 100 hours is rarely what moves a model. Going from 50 to 800 might be.
- Matched data beats more data. If extra hours stop helping, collecting different audio (new speakers, new conditions, new vocabulary) usually does more than collecting more of the same.
Build the test set before the training set
You can't size a training set without a way to measure it, so the first audio you should pin down is the audio you'll never train on.
- Use different speakers. Nobody in the test set should appear in training, or the scores will flatter the model.
- Record it in your real conditions. If users call in from cars on cheap phones, the test set should too.
- Cover each group you report on. If you need results by accent, region or age band, each group needs its own share of test audio.
- Normalize text before scoring. Differences in punctuation, casing and number formats can swing WER without any real change in accuracy. OpenAI released the text normalizer it used for Whisper's own evaluations for exactly this reason.
A few hours is usually enough to compare runs with confidence. Keep it frozen. If the test set changes between runs, you can't tell whether the model got better or the test got easier.
Rough starting points by goal
With all of that said, teams still need a number to plan around. These are the ranges we'd suggest a first pilot aims for. They're planning ranges, not results: the test set decides where you end up.
Where to start a pilot, by goal
Hours of transcribed speech, on a log scale. Planning ranges, not results.
- Domain words or one accent, in a language Whisper knows well10 to 50 h
- Noisy channels, far-field audio or a wider accent mix50 to 200 h
- A language Whisper barely covers200 to 1,000 h
- Training a recognition model from scratch, one language1,000 h and up
Two benchmarks help put those ranges in context. LibriSpeech, the classic English recognition benchmark, is about 1,000 hours of read audiobook speech, and plenty of models have been trained on it from scratch. At the other extreme, wav2vec 2.0 reached 4.8 and 8.2 WER on LibriSpeech's clean and other test sets with just ten minutes of labeled audio, after self-supervised pretraining on 53,000 hours of unlabeled speech. Large amounts of untranscribed audio can stand in for some transcribed hours, if your model and pipeline are built for it. Whisper fine-tuning, though, needs audio with transcripts.
What counts as much as the hours
Two datasets with the same number of hours can produce very different models. Before you compare quotes by the hour, compare these:
How many people are speaking
An hour from one speaker teaches the model one voice. An hour spread across a hundred speakers teaches it how a language varies. Ask how many speakers a dataset has, and the most any one of them contributes.
Whether the conditions match yours
Studio-quality read speech won't prepare a model for a noisy call center line or a smart speaker across the room. Match the microphone, the channel, the background noise and the distance to what your product meets.
How consistent the transcripts are
If one transcriber writes "two hundred" and another writes "200", or one marks filler words and another drops them, the model learns noise. Agree on a style before transcription starts, including how to handle code-switching between languages. Our guide to transcription for AI covers the choices.
Whether your words are in it
Product names, medical terms and street names that never appear in the audio won't be recognized well, however many hours you have. Scripted prompts are the reliable way to cover them.
About 35,000 hours across 71 languages, already recorded and transcribed, with sample clips on request.
Buy in steps, not all at once
The cheapest way to find your number is to measure it. A simple plan:
- Fine-tune a pilot. Start at the low end of your range from the chart above, and score it on the frozen test set.
- Double, then measure again. Add data in big steps, such as 2x or 4x. Small steps get lost in the noise between runs.
- Plot WER against hours on a log scale. If the points sit roughly on a line, extend it to your target WER to estimate the hours you need.
- Watch for the curve flattening. If extra hours stop helping, more of the same audio won't fix it. Look at the errors and collect what's missing: speakers, conditions or vocabulary.
Buying in steps also keeps options open. The pilot might show that an off-the-shelf dataset gets you most of the way and only a small custom top-up is needed. We cover that trade-off in off-the-shelf or custom speech data: how to choose.
Where the hours come from
There are three places to get transcribed speech, and most projects use more than one.
- Public corpora. Mozilla Common Voice covers many languages with crowd-read sentences and is released under CC0, though newer downloads through Mozilla add terms such as not re-sharing the data. LibriSpeech is CC BY 4.0. Both are great for getting started. Check the licence of anything else before you train a product on it, and remember that read sentences don't sound like real conversations.
- Off-the-shelf datasets. Audio that's already recorded and transcribed, licensed per project. Our OTS pool holds about 35,000 hours across 71 languages, roughly 500 hours per language and about 1,000 for Hindi, and every speaker consented to commercial AI training.
- Custom collection. Speakers, conditions and prompts chosen to your brief, for the gaps nothing else covers. Our speech collection brief template lists what to decide first.
If you tell us the language, the use case and the error rate you're aiming for, we'll help you size the first step and send sample clips so you can test before you commit.
