Existing benchmarks suggest that voice AI performance is approaching human levels, but real-world conversations tell a different story.
Voice is rapidly becoming the primary interface for AI. From customer support and healthcare to education, entertainment, and personal assistants, voice is increasingly replacing text as the way people interact with AI.
Over the past few years, voice models have improved dramatically. Word error rates continue to decline, latencies reach conversational speeds, and many established benchmarks approach saturation. But anyone who uses voice AI regularly knows that something still feels off.
Voice models can sound like someone else during a conversation, miss hesitations and uncertainties, and struggle with accents, noises, and emotional speaking. These drawbacks are often overlooked in benchmarks that focus on latency and word error rates. People are concerned about whether their voice systems can truly hear, respond appropriately, and remain natural and believable in real-life conversations.
Extensive benchmarks for voice AI
To measure these qualities, we built Real World VoiceEQ, a benchmark designed to assess the human quality of voice interaction. Assess whether speech systems can recognize, generate, and respond to stranded acoustic information transcripts, from tone and emotion to speaker identity and background context.
Real World VoiceEQ evaluates over 40 leading proprietary open-source speech models across over 15 key evaluation dimensions and over 60 metrics spanning Automatic Speech Recognition (ASR), Text-to-Speech (TTS), Text-to-Speech (S2S), and Speech Understanding.
Real World VoiceEQ was developed based on ratings of over 1 million individuals across a variety of demographics, speaking styles, and acoustic environments. The current benchmark includes 785,000 TTS ratings and 48,000 STS ratings, making it one of the largest human evaluations of voice AI ever conducted.
All evaluations were conducted using Kairos, our flexible voice-native evaluation platform. The same infrastructure enables leading AI labs and enterprises to perform custom assessments tailored to specific use cases, identify detailed failure modes in production audio systems, generate human preference data, and continuously improve models through reinforcement learning and human feedback.
Key findings from Real World VoiceEQ
Advances in voice AI are becoming increasingly specialized.
The race for a single “best” voice model is giving way to a collection of specialized features.
Today’s leading systems optimize a variety of strengths, including technical accuracy, emotional understanding, conversational intelligence, expressiveness, and robustness. A model that excels at repeating booking reference numbers, bank account details, or complex drug names may struggle to produce emotionally expressive speech. The other may sound very natural, but is less reliable for accuracy-oriented tasks.
As voice AI matures, measuring progress will increasingly require evaluating these capabilities individually rather than combining them into one overall score. In our TTS evaluation, no system configuration ranked in the top five in all eight functional groups. This highlights why there is no single “best” speech model.
Voice models have become better at speaking than actually listening.
The Speech-to-Speech model showed the widest variation of the categories evaluated. Some systems recognized emotions very well, but struggled to respond naturally. We found that access to audio does not guarantee that agents use the quasi-linguistic information contained in the audio. Some systems were still primarily transcript-driven, relying on the words spoken and ignoring cues such as tone, pace, hesitation, emphasis, and volume.
Humans naturally use these cues to infer confidence, uncertainty, frustration, cynicism, and empathy. Today’s models often miss them.
Imagine a bank agent asks you if you are aware of a potentially fraudulent transaction. Even if the transcription is the same, a confident “yes” and a hesitant “…yes…” can have completely different meanings. Humans quickly recognize the difference. Many of today’s voice models do not.
Traditional benchmarks increasingly overestimate real-world performance.
Many established benchmarks are marginal and do not reflect real-world conditions. Models still suffer from accented voices, duplicate speakers, emotions, background noise, and long conversations. In our evaluation, there was greater variation in performance between leading open source and proprietary models than traditional benchmarks indicate. In one example, the error rate for transcribed words for speech against a background of noise was about four times higher than for speech against a background of music. This shows how a single background audio score can hide the actual failure mode.
Human evaluation remains important.
In our preliminary research, we found indications that some models may be optimized against established public benchmarks. Some reproduced known errors in reference transcripts, followed arbitrary spelling rules, and even reconstructed masked words that were not present in the audio.
Although LLMs are currently widely used to evaluate text-based models, our findings suggest that speech language models (SLMs) should be used more carefully for speech evaluation. When comparing leading SLMs and trained human raters for text-to-speech assessments, agreement was highest for tasks that yield clear and verifiable answers, such as pronunciation accuracy.
More subjective evaluations showed less agreement. Although SLM sometimes inferred emotion from text-based contextual cues, agreement was weakest for free-form judgments, such as whether a voice fits the acting role or maintains a consistent identity. Although automated evaluators are useful for well-defined tasks, they still cannot replace human listeners when judgments depend on acoustic context, perception, and social interpretation.
Why Voice AI needs a new measurement layer
As voice becomes one of the interfaces that define AI, speed and technical accuracy alone will no longer determine which systems are successful. The model that people ultimately choose will be one that can understand, express, and respond like a human, not just under ideal benchmark conditions but also in complex real-world conversations.
For decades, voice AI has advanced by optimizing against quantitative metrics in standardized benchmarks. From WER, which measures transcription accuracy, to objective perceptual metrics such as PESQ and DNSMOS, which measure audio quality. We hope that Real World VoiceEQ can extend this paradigm by providing human-based metrics for evaluating the components of synthetic voice interactions.
Read the full technical report and explore public leaderboards. Or, if you would like to learn more about how Hume uses Real World VoiceEQ to evaluate voice models or agents, or how we can design custom evaluations for your specific use case, please contact us.
