Gated Embeddings in End-to-End Speech Recognition for Conversational-Context Fusion

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1131–1141 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1131 Gated Embeddings in End-to-End Speech Recognition for Conversational-Context Fusion Suyoun Kim1, Siddharth Dalmia2 and Florian Metze2 1Electrical & Computer Engineering 2Language Technologies Institute, School of Computer Science Carnegie Mellon University {suyoung1, sdalmia, fmetze}@andrew.cmu.edu Abstract We present a novel conversational-context aware end-to-end speech recognizer based on a gated neural network that incorpo- rates conversational-context/word/speech em- beddings. Unlike conventional speech recog- nition models, our model learns longer conversational-context information that spans across sentences and is consequently better at recognizing long conversations. Specifi- cally, we propose to use text-based external word and/or sentence embeddings (i.e., fast- Text, BERT) within an end-to-end framework, yielding significant improvement in word er- ror rate with better conversational-context rep- resentation. We evaluated the models on the Switchboard conversational speech corpus and show that our model outperforms standard end-to-end speech recognition models. 1 Introduction In a long conversation, there exists a tendency of semantically related words, or phrases reoccur across sentences, or there exists topical coherence. Existing speech recognition systems are built at in- dividual, isolated utterance level in order to make building systems computationally feasible. How- ever, this may lose important conversational con- text information. There have been many studies that have attempted to inject a longer context infor- mation (Mikolov et al., 2010; Mikolov and Zweig, 2012; Wang and Cho, 2016; Ji et al., 2016; Liu and Lane, 2017; Xiong et al., 2018), all of these mod- els are developed on text data for language model- ing task. There has been recent work attempted to use the conversational-context information within a end-to-end speech recognition framework (Kim and Metze, 2018; Kim et al., 2018; Kim and Metze, 2019). The new end-to-end speech recog- nition approach (Graves et al., 2006; Graves and Jaitly, 2014; Hannun et al., 2014; Miao et al., 2015; Bahdanau et al., 2015; Chorowski et al., 2015; Chan et al., 2016; Kim et al., 2017) in- tegrates all available information within a sin- gle neural network model, allows to make fus- ing conversational-context information possible. However, these are limited to encode only one pre- ceding utterance and learn from a few hundred hours of annotated speech corpus, leading to min- imal improvements. Meanwhile, neural language models, such as fastText (Bojanowski et al., 2017; Joulin et al., 2017, 2016), ELMo (Peters et al., 2018), OpenAI GPT (Radford et al., 2019), and Bidirectional En- coder Representations from Transformers (BERT) (Devlin et al., 2019), that encode words and sen- tences in fixed-length dense vectors, embeddings, have achieved impressive results on various nat- ural language processing tasks. Such general word/sentence embeddings learned on large text corpora (i.e., Wikipedia) has been used exten- sively and plugged in a variety of downstream tasks, such as question-answering and natural lan- guage inference, (Devlin et al., 2019; Peters et al., 2018; Seo et al., 2017), to drastically improve their performance in the form of transfer learning. …

Content truncated to 500 words