How Accents Confound: Probing for Accent Information in End-to-End Speech Recognition Systems

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3739–3753 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3739 How Accents Confound: Probing for Accent Information in End-to-End Speech Recognition Systems Archiki Prasad Department of EE, IIT Bombay, India archikiprasad@gmail.com Preethi Jyothi Department of CSE, IIT Bombay, India pjyothi@cse.iitb.ac.in Abstract In this work, we present a detailed analysis of how accent information is reflected in the in- ternal representation of speech in an end-to- end automatic speech recognition (ASR) sys- tem. We use a state-of-the-art end-to-end ASR system, comprising convolutional and recur- rent layers, that is trained on a large amount of US-accented English speech and evaluate the model on speech samples from seven different English accents. We examine the effects of ac- cent on the internal representation using three main probing techniques: a) Gradient-based explanation methods, b) Information-theoretic measures, and c) Outputs of accent and phone classifiers. We find different accents exhibit- ing similar trends irrespective of the probing technique used. We also find that most ac- cent information is encoded within the first re- current layer, which is suggestive of how one could adapt such an end-to-end model to learn representations that are invariant to accents. 1 Introduction Traditional automatic speech recognition (ASR) systems, consisting of independently-trained acous- tic, pronunciation and language models, are in- creasingly being replaced by end-to-end ASR sys- tems (Chiu et al., 2018; Hori et al., 2017). An end-to-end ASR system refers to a single model that subsumes all the traditional ASR components and directly translates a speech utterance into a se- quence of graphemes. Such models benefit from jointly training acoustic and language models and eliminating the need for a pronunciation dictionary. While end-to-end ASR models have clear merits and are elegant in their formulation, they tend to be opaque in their predictions and difficult to interpret. In order to understand better what is encoded in the layers of an end-to-end ASR system, prior work has explored the use of phone probes (classifiers) to analyze the phonetic content of representations at each layer (Belinkov and Glass, 2017; Belinkov et al., 2019). This analysis was restricted to a sin- gle accent of English. In this paper, we work with multiple accents of English and propose a number of different tools (other than phone probes) to in- vestigate how accent information is encoded and propagated within an end-to-end ASR system. Why accented speech? We have witnessed im- pressive strides in ASR performance in the last few years. However, recognizing heavily accented speech still remains a challenge for state-of-the-art ASR systems. An end-to-end ASR model trained on a standard speech accent significantly underper- forms when confronted with a new speech accent. To shed more light on why this happens, a sys- tematic investigation of how such models behave when evaluated on accented speech might be useful. The insights from such an investigation might also come in handy when trying to adapt end-to-end neural architectures to be more accent-agnostic. We tackle …

Content truncated to 500 words