Language technology practitioners as language managers: arbitrating data bias and predictive bias in ASR

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6328–6339 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6328 Language technology practitioners as language managers: arbitrating data bias and predictive bias in ASR Nina Markl, Stephen Joseph McNulty University of Edinburgh nina.markl@ed.ac.uk, stephen.mcnulty@ed.ac.uk Abstract Despite the fact that variation is a fundamental characteristic of natural language, automatic speech recognition systems perform systematically worse on non-standardised and marginalised language varieties. In this paper we use the lens of language policy to analyse how current practices in training and testing ASR systems in industry lead to the data bias giving rise to these systematic error differences. We believe that this is a useful perspective for speech and language technology practitioners to understand the origins and harms of algorithmic bias, and how they can mitigate it. We also propose a re-framing of language resources as (public) infrastructure which should not solely be designed for markets, but for, and with meaningful cooperation of, speech communities. Keywords: Automatic Speech Recognition, Algorithmic Bias, Language Policy 1. Introduction All language communities, even monolingual ones, show linguistic variation. The co-existence of multiple different ways of communicating the same meaning is a fundamental characteristic of natural language. Speak- ers can employ different words to refer to the same ob- ject (e.g. “film” and “movie”), pronounce the same word differently (e.g. “data” and “data”), address in- terlocutors differentially depending on the context (e.g. “you”, “yous”, “y’all” in many varieties of English), and even utilise different sentence structures (e.g. “data is” and “data are”). Despite that fact that this varia- tion is inevitable, people still form judgements about them. Very often, these judgements reflect biases about (groups of) people, not language per se. These social-linguistic judgements contribute to differ- ential access to, and performance of, language tech- nologies for speakers of the over 7000 language va- rieties1 spoken in the world. Most language com- munities globally do not have access to them at all, and within those that do, performance for speakers of non-standard(ised) and marginal(ised) varieties is worse. For automatic speech recognition (ASR) sys- tems, this “predictive bias”, defined by Shah et al. (2020) as a systematic error disparity between differ- ent user groups, arises in part from data bias in the speech datasets used to train and test them. In this pa- per, we use the lens of “language policy” to understand the origins and consequences of this data bias, and to facilitate its mitigation. We contend that, perhaps un- knowingly, organisations – and particularly individuals – involved in the design and creation of these datasets, whether crowdsourced or curated, perform the func- tion of “language policy arbiters” (Johnson, 2013). In 1“Language variety” refers to languages (e.g. English), “dialects” (e.g. Scottish English) and accents (e.g. Standard Scottish English). The linguistic features characterising a va- riety are called “variants”. their selection of widely spoken, prestigious (and often commercially-viable) varieties, these individuals effec- tively marginalise speakers of minority or lesser-used languages or …

Content truncated to 500 words