SPLAT: Speech-Language Joint Pre-Training for Spoken Language Understanding

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1897–1907 June 6–11, 2021. ©2021 Association for Computational Linguistics 1897 SPLAT: Speech-Language Joint Pre-Training for Spoken Language Understanding Yu-An Chung1⇤, Chenguang Zhu2⇤, Michael Zeng2 1MIT Computer Science and Artificial Intelligence Laboratory 2Microsoft Cognitive Services Group andyyuan@mit.edu,{chezhu,nzeng}@microsoft.com Abstract Spoken language understanding (SLU) re- quires a model to analyze input acoustic sig- nal to understand its linguistic content and make predictions. To boost the models’ per- formance, various pre-training methods have been proposed to learn rich representations from large-scale unannotated speech and text. However, the inherent disparities between the two modalities necessitate a mutual analy- sis. In this paper, we propose a novel semi- supervised learning framework, SPLAT, to jointly pre-train the speech and language mod- ules. Besides conducting a self-supervised masked language modeling task on the two in- dividual modules using unpaired speech and text, SPLAT aligns representations from the two modules in a shared latent space using a small amount of paired speech and text. Thus, during fine-tuning, the speech module alone can produce representations carrying both acoustic information and contextual se- mantic knowledge of an input acoustic signal. Experimental results verify the effectiveness of our approach on various SLU tasks. For example, SPLAT improves the previous state- of-the-art performance on the Spoken SQuAD dataset by more than 10%. 1 Introduction Spoken language understanding (SLU) tackles the problem of comprehending audio signals and mak- ing predictions related to the content. SLU has been widely employed in various areas such as intent understanding (Tur and De Mori, 2011; Bhargava et al., 2013; Ravuri and Stolcke, 2015; Lugosch et al., 2019), question answering (Lee et al., 2018; Chuang et al., 2020), and sentiment analysis (Zadeh et al., 2018). Early approaches leverage a two-step pipeline: use automatic speech recognition (ASR) to transcribe input audio into text, and then em- ploy language understanding models to produce ⇤ Equal contribution. The work was done when Yu-An Chung was interning at Microsoft. results. However, such cascaded system has sev- eral drawbacks. First, the transcription produced by the ASR module often contains errors, which adversely affects the language understanding mod- ule’s prediction accuracy. Second, even if the tran- scription is perfect, the rich prosodic information of speech (e.g., tempo, pitch, and intonation) is in- evitably lost after ASR. In comparison, humans of- ten leverage these information to better understand and disambiguate the content. Therefore, there has been a rising trend of end-to-end approaches to retain information from audio signals to carry out the understanding task (Serdyuk et al., 2018; Chen et al., 2018; Haghani et al., 2018). While end-to-end SLU methods are effective, they often suffer from a shortage of labeled training data, especially when the target task is in a novel domain. One solution is to leverage self-supervised training as is done in pre-trained language mod- els. Examples like BERT (Devlin et al., 2019), GPT (Radford et al., 2018), and RoBERTa (Liu et al., 2019) are …

Content truncated to 500 words