🚀 First open far-field ASR benchmark: Community-driven evaluation across 14 simulated rooms, validated against real-world measurements: https://huggingface.co/spaces/treble-technologies/ffasr
📉 This gap is real, and it’s big. Across all submitted models, the far-field WER at low SNR is consistently several times higher than the near-field WER for the same audio content.
🔬 Trusted methodology: hybrid waveform-based simulation, sim-to-real validation, beta moving source split, holdout audio, and standardized evaluation hardware across all submissions
⚡ Accuracy meets speed: Pareto front plots plot the average WER against RTFx, allowing you to evaluate trade-offs for your deployment.
đź‘€ More coming: Multi-talker scenarios, microphone array support, echo cancellation on the roadmap
The gap between benchmark performance and real-world deployment is one of the most persistent frustrations in ASR development. Models that score well in standard evaluations often behave differently when real room acoustics (reverberation, background noise, microphone distance) are involved. The complex interactions between these factors affect performance in ways that clean speech benchmarks cannot capture. The FFASR Leaderboard is an attempt to quantify that gap.
Treble Technologies and Hugging Face launch the Far-Field ASR (FFASR) Leaderboard, the first open, community-driven benchmark designed to evaluate ASR models under realistic far-field acoustic conditions. It’s open now and invites the community to submit models, explore results, and help shape what comes next.
Why far-field evaluation is important
Voice interfaces have expanded far beyond headsets and smartphones. AI voice agents, conference room transcription, in-vehicle assistants, humanoid robots, smart glasses, and hands-free tools are all rapidly becoming popular. What they all have in common is that they operate in acoustically complex environments, including reverberation, background noise, overlapping sounds, and microphones located one to several meters away from the loudspeaker.
The dominant ASR evaluation paradigm has not kept up with this reality. Clean, near-mic benchmarks are still the standard, and while they are useful for measuring core recognition quality, they are not predictive of far-field performance. Models that work well in LibriSpeech and other near-field sets can perform significantly worse when real-world room acoustics enter. Although there has been some research work on the evaluation of far-field and noisy speech, including CHiME, URGENT, and NOIZEUS, the community lacked a standardized and open method to consistently measure its degradation across models in the form of a continuously updated leaderboard. That’s what FFASR was built for.
The main challenge for far-field evaluation is data availability. Physical measurements alone are prohibitively expensive to collect large-scale far-field recordings over a representative range of room types, microphone distances, and noise conditions. Simulation makes it possible to systematically cover that space and increase coverage over time without commensurately increasing measurement costs.
Another goal of FFASR is to encourage the development of models that are explicitly robust to these conditions. Leaderboards have historically been effective in directing research efforts. By enabling visualization and comparison of far-field performance, we hope to prioritize real-world acoustic robustness across the field.
How to build a benchmark
The FFASR leaderboard evaluates models across nine conditions. There are four things that determine the main ranking score (as of June 22, 2026):
Near field (dry) — Clean audio measured in an anechoic chamber (similar to Librispeech, but with minimal reverberation) Far range high SNR (>14 dB) Far range medium SNR (8-12 dB) Far range low SNR (<6 dB)
To understand what these situations actually sound like, in the sample below we listen to the same utterance as anechoic anechoic audio, then convolved with the room impulse response, and finally with noise added at each SNR layer. The difference between dry recordings and low SNR far-field conditions is a good representation of the magnitude of the problem that the leaderboard is measuring.
Two additional columns (Lab Measured and Lab Simulated) serve as simulation-to-real validation tracks. The leaderboard also includes a moving source split, currently in beta, which evaluates models for audio where the speakers are moving rather than stationary. This condition reflects use cases such as humanoid robots, in-vehicle speech recognition, and mobile voice assistants, where the acoustic geometry between speaker and microphone changes continuously.
Acoustic data is generated by Treble’s hybrid simulation engine. The engine combines a wave-based solver at low to mid frequencies with geometric acoustic modeling at high frequencies. This approach captures physical phenomena that are often missed by simple simulation methods, such as diffraction, scattering, interference, and modal behavior. The results are simulated data that closely match the measured acoustic conditions, which is directly confirmed by performing the same evaluation on both the lab-measured and lab-simulated columns.
The benchmark includes 14 furnished rooms ranging from 20 to 470 m3, including a bathroom, living room with hallway, office, classroom and restaurant space. Each acoustic scene contains one target speaker and up to three noise sources, which were recorded in an anechoic chamber to avoid reverberation artifacts from the recording environment. All scenes contain both transient noise sources such as coughing and continuous noise sources such as HVAC at three SNR levels. This coverage is designed to reflect the variety of real-world spaces in which deployed audio systems operate.
Along with WER, the leaderboard reports RTFx (audio seconds per inference) for all submissions evaluated on NVIDIA L4 GPUs under the same conditions. Both accuracy and latency are important in real-world deployments, and the Pareto front view in the (Analysis) tab makes the trade-offs clear.
This benchmark is built on a simulated acoustic space through Treble Technologies’ proprietary simulation engine. An example of the output from the engine can be found in the Treble10 dataset released last year. This established a simulation pipeline and made far-field RIR available for training and research. FFASR extends its foundation to a standardized assessment framework with preserved test sets, consistent normalization, and automated scoring.
What the data already shows
The real-time leaderboard shows a consistent pattern across all submitted models. In other words, the difference between near-field and far-field performance is large and increases as the SNR decreases. Near-field WER values ​​with clean, dry audio appear comparable to those achieved by the same model on established benchmarks. Far-field WER at low SNR presents a different picture and is often several times higher. This benchmark allows us to visualize and compare this degradation in a way that has previously been difficult to do outside of proprietary evaluation pipelines.
The Pareto front of average WER to RTFx is also evident. Current applications include a variety of approaches. Some models prioritize speed at the expense of accuracy, others increase accuracy at the expense of throughput, and a small number of models achieve competitive positions on both axes. Visualizing these trade-offs for far-field accuracy rather than clean speech accuracy gives a much different picture of where the real differences between systems lie. The ‘Analysis’ tab is also worth considering beyond the main leaderboard.
There is one observation worth noting for developers. The leaderboard reports both near field (dry) and far field WER side by side. This separation is intentional and beneficial. This makes it possible to distinguish between truly accurate models and those that are accurate but vulnerable to acoustic conditions. This is important in deciding whether to invest in far-field fine-tuning, audio enhancement pre-processing, or a completely different architecture.
How to submit
Open the (Submit) tab of the FFASR Leaderboard, paste the Hugging Face model ID, and the evaluation will be performed server-side against the persisted dataset. This pipeline supports Whisper variants, IBM Granite Speech, Cohere Transcribe, Wav2Vec2 and HuBERT CTC heads, SpeechBrain ASR, and most other ASR architectures on the hub without any custom configuration.
For teams using more complex inference stacks, such as systems that combine speech enhancement and ASR, you can define your own Evaluator() function using the custom evaluator option. Custom evaluators are run in a hub job after moderator review. The submission notes field is suitable for documenting preprocessing steps so that the results can be interpreted by others.
The held-out evaluation set uses 2,000 anechoic audio samples across 14 rooms in three SNR layers, approximately 8 hours of audio per condition, and consistently applied Whisper-style text normalization. To avoid contaminating the test set, the audio will not be made public to submitters.
what comes next
Conditions we are actively considering for future tracks include multi-talker scenarios where multiple speakers are active simultaneously, evaluation of microphone arrays covering beamforming and spatial filtering approaches, and echo cancellation associated with any device that plays audio while listening.
What we build next will depend on what the community tells us where the gaps are greatest. If you’re working on a deployment environment or use case that isn’t well represented in current benchmarks, we’d love to hear from you. FFASR leaderboards are designed to grow, and the direction in which you grow should reflect real-world needs.
Submit your model, explore the (Analysis) tab, and post your ideas and suggestions on the FFASR forum to help build benchmarks that actually help address the problems the field is tackling.
