Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and Accented Speech

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6751–6760 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6751 Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and Accented Speech Katrin Tomanek∗, Vicky Zayats∗, Dirk Padfield, Kara Vaillancourt, Fadi Biadsy Google, USA {katrintomanek,vzayats,padfield,kvailla,biadsy}@google.com Abstract Automatic Speech Recognition (ASR) sys- tems are often optimized to work best for speakers with canonical speech patterns. Un- fortunately, these systems perform poorly when tested on atypical speech and heavily ac- cented speech. It has previously been shown that personalization through model fine-tuning substantially improves performance. However, maintaining such large models per speaker is costly and difficult to scale. We show that by adding a relatively small number of ex- tra parameters to the encoder layers via so- called residual adapter, we can achieve simi- lar adaptation gains compared to model fine- tuning, while only updating a tiny fraction (less than 0.5%) of the model parameters. We demonstrate this on two speech adaptation tasks (atypical and accented speech) and for two state-of-the-art ASR architectures. 1 Introduction Automatic Speech Recognition (ASR) systems have achieved great success on a diverse set of acoustic and linguistic conditions, domains and speech patterns. State-of-the-art ASR systems are typically trained on tens of thousands of hours of speech data, and they perform well as long as these domains and conditions are well represented in the training data. Understandably, the distribution of such data typically focuses on the canonical and typical spo- ken language patterns of the target language, i.e., regional dialects, common accents and frequent non-native accents. As a result, these systems may perform poorly on the tail of the distribu- tion which may include “heavily” accented speech and/or speech with atypical speech patterns (Darley et al., 1975). Atypical speech includes dysarthric speech, speech impairments (due to, for example, ALS, stroke, traumatic brain injury, down syn- drome, cerebral palsy, and MS), stuttering, deaf ∗ Equal contribution speech, or severe hyper-nasality due to cleft lip and palate. The lack of sufficient training data for these accents and atypical speech in the training distribution may result in a poor experience for a large segment of the population, leaving the less fortunate communities behind when it comes to speech-enabled technologies (Moore et al., 2018). Studies on accented speech showed word error rates (WER) twice or three times as high for ac- cented speech compared to the more standard US accent (Sainath et al., 2020; Ghorbani and Hansen, 2018). Even worse performance is observed for speakers with speech impairments (Moore et al., 2018). Our goal, in this paper, is to efficiently build scalable models that can adapt to non-canonical or atypical speech. It has been shown that speech models originally developed for typical speech can be successfully fine-tuned with limited amounts of data to accented or impaired speech (Zhu et al., 2019; Shor et al., 2019; Gale et al., 2019; Mustafa et al., 2014; Biadsy et al., 2019; Doshi et al., 2020; Green et al., 2021). Nevertheless, one major challenge with …

Content truncated to 500 words