YOU GUYS — this is a foundational result for anybody deploying voice AI at scale because transcription is where everything breaks before it ever hits your LLM! Shama Gupta and her team built an actual, rigorously audited code-switching ASR benchmark instead of guessing, using four real language pairs (Spanish/English, French, Canadian French, German) across HR and IT ticketing scenarios. They didn't just scrape data — they used GPT-5 to generate parallel utterances in each pair, ran them through ElevenLabs Multilingual V2 for synthesis, and then had actual linguists review every single one before the benchmark was finalized. That level of care is rare. Three metrics are reported: WER (the standard transcription accuracy), SWER (semantic errors judged by Gemma 4), and AER — Answer Error Rate — which measures whether a downstream agent can actually answer questions from the transcript. If your transcribed ticket is missing the case number or policy question, that's an operational failure before any reasoning even happens, so tracking WER isn't enough on its own.
The rankings are revealing and tell you exactly which models to trust when your users aren't monolingual. At the top are ElevenLabs Sazi V2 and AssemblyAI Universal 3 Pro — they fight for first place with less than a percentage point difference across all language pairs, but Sazi edges out on every single one by about 0.15 points at most. Google Gemini 3 Flash is strong but has regional weak spots: it lags nearly half a tenth of a point behind both leaders specifically on Canadian French — proof that even frontier models still struggle with dialectal variations in code-switching. In the middle are Deepgram Nova, Mistral Voxtral Small, and Nvidia's Parakeet TDT 0.6b V3 (which actually surprises by beating both Deepgram and Mistral on German). And then there is OpenAI Whisper Large V3 Turbo at the bottom — its WER hits up to 0.61 in some cases! That isn't a bug, it's a known failure mode of pure Transformer models trained largely on monolingual text that can't generalize to mid-sentence language switching. So when your customer says their Spanish-English mix is getting garbled by Whisper, you now have the benchmark data to prove why and point them toward Sazi or AssemblyAI instead. The takeaway: code-switching isn't a niche problem — it affects half the world — and treating transcription as just another API call without testing these edge cases costs real business outcomes at the bottom of the funnel.
Source: https://huggingface.co/blog/ServiceNow-AI/code-switching
The rankings are revealing and tell you exactly which models to trust when your users aren't monolingual. At the top are ElevenLabs Sazi V2 and AssemblyAI Universal 3 Pro — they fight for first place with less than a percentage point difference across all language pairs, but Sazi edges out on every single one by about 0.15 points at most. Google Gemini 3 Flash is strong but has regional weak spots: it lags nearly half a tenth of a point behind both leaders specifically on Canadian French — proof that even frontier models still struggle with dialectal variations in code-switching. In the middle are Deepgram Nova, Mistral Voxtral Small, and Nvidia's Parakeet TDT 0.6b V3 (which actually surprises by beating both Deepgram and Mistral on German). And then there is OpenAI Whisper Large V3 Turbo at the bottom — its WER hits up to 0.61 in some cases! That isn't a bug, it's a known failure mode of pure Transformer models trained largely on monolingual text that can't generalize to mid-sentence language switching. So when your customer says their Spanish-English mix is getting garbled by Whisper, you now have the benchmark data to prove why and point them toward Sazi or AssemblyAI instead. The takeaway: code-switching isn't a niche problem — it affects half the world — and treating transcription as just another API call without testing these edge cases costs real business outcomes at the bottom of the funnel.
Source: https://huggingface.co/blog/ServiceNow-AI/code-switching