Open ASR Leaderboard Adds Hindi and Indian English to Address Bias in Speech Recognition

Addressing ASR Bias with New Language Additions

The Open ASR Leaderboard has introduced two new evaluation sets, Monsoon en-IN and Monsoon hi-IN, to incorporate Hindi and Indian English. This marks the first inclusion of languages from the Global South on the leaderboard, which previously focused on European languages. The move is a direct response to documented biases in automatic speech recognition (ASR) systems, which exhibit varying performance across different demographics and accents.

The Importance of Diverse Benchmarks

Existing ASR benchmarks often fail to capture performance disparities across diverse user groups, such as differences based on race, gender, age, and accent. While the Open ASR Leaderboard has improved its evaluation metrics to prevent benchmark-fitting, the underlying test sets historically lacked demographic information. The addition of Monsoon en-IN and Monsoon hi-IN aims to rectify this by providing data that varies along nine axes, including geography, age, gender, and acoustic environments.

Monsoon Dataset Design and Collection

The Monsoon datasets are designed to expose failure modes that aggregate Word Error Rate (WER) metrics might otherwise obscure. Each set includes both public and private splits and is speaker-disjoint, comprising 4,888 speakers with 12 recorded attributes. The collection methodology emphasized diversity, recruiting participants from hundreds of districts and having them use their own devices in various acoustic conditions to ensure a realistic representation of speech variations.

Impact on ASR Development

By expanding the leaderboard to include more diverse languages and speaker attributes, the initiative encourages the development of more equitable and robust ASR models. Benchmarks significantly influence the direction of model development, and the inclusion of these new datasets is expected to drive improvements in ASR performance for a broader global population, particularly for speakers of Indic languages.