Attention-based Contextual Language Model Adaptation for Speech Recognition

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1994–2003 August 1–6, 2021. ©2021 Association for Computational Linguistics 1994 Attention-based Contextual Language Model Adaptation for Speech Recognition Richard Diehl Martinez, Scott Novotney, Ivan Bulyko, Ariya Rastrow, Andreas Stolcke, Ankur Gandhe Amazon Alexa, Seattle, WA, USA {mrtimri,snovotne,ibbulyko,arastrow,stolcke,aggandhe}@amazon.com Abstract Language modeling (LM) for automatic speech recognition (ASR) does not usually in- corporate utterance level contextual informa- tion. For some domains like voice assistants, however, additional context, such as time at which an utterance was spoken, provides a rich input signal. We introduce an attention mechanism for training neural speech recog- nition language models on both text and non- linguistic contextual data 1. When applied to a large de-identified dataset of utterances col- lected by a popular voice assistant platform, our method reduces perplexity by 7.0% rela- tive over a standard LM that does not incorpo- rate contextual information. When evaluated on utterances extracted from the long tail of the dataset, our method improves perplexity by 9.0% relative over a standard LM and by over 2.8% relative when compared to a state-of-the- art model for contextual LM. 1 Introduction Conventional automatic speech recognition (ASR) systems include a language model (LM) and an acoustic model (AM). The LM component is trained separately, typically on large amounts of transcribed utterances that have been collected by an existing speech recognition system. Voice assistants have become ubiquitous and cru- cially rely on ASR systems to convert user inputs to text. They often collect utterances spoken by users, along with associated de-identified contextual in- formation. We hypothesize that these additional data, such as the time at which an utterance was spoken, provide a useful input signal for a LM. As an example, knowing that an utterance was spoken on December 25th, a LM should learn that the word “christmas” rather than “easter” is more likely to follow the phrase “lookup cookie recipes for”. 1We make a large portion of our code available un- der: https://github.com/amazon-research/contextual-attention- nlm To-date, some voice assistants have leveraged coarse geographic information for improving loca- tion search queries (Bocchieri and Caseiro, 2010; Lloyd and Kristjansson, 2012). These past efforts, however, have largely focused on improving a par- ticular skill of an ASR system, and not the system’s speech recognition in general. In this paper, we focus on adapting recurrent neural network language models (RNN-LMs) to use both text and non-linguistic contextual data for speech recognition in general. While, outside of ASR, transformer-based (Vaswani et al., 2017) language models have largely replaced RNN-LMs, RNNs remain dominant in ASR architectures such as connectionist temporal classification (Graves et al., 2006), and RNN-T (Graves, 2012; He et al., 2019). The most common method for incorporating non- linguistic information into a RNN-LM is to learn a representation of the context that is concatenated with word embeddings as input to the model. This concatenation-based approach has been used in a variety of domains including text classification (Yo- gatama et al., 2017), personalized conversational agents (Wen et al., 2013), and voice …

Content truncated to 500 words