Speech-Text Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignment

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7900–7913 July 9-14, 2023 ©2023 Association for Computational Linguistics Speech-Text Dialog Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignment Tianshu Yu123∗, Haoyu Gao134∗, Ting-En Lin3, Min Yang1†, Yuchuan Wu3, Wentao Ma3, Chao Wang3, Fei Huang3, Yongbin Li3† 1Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences 2University of Chinese Academy of Sciences 3Alibaba Group 4University of Science and Technology of China {ts.yu, hy.gao, min.yang}@siat.ac.cn {ting-en.lte,shuide.lyb}@alibaba-inc.com Abstract Recently, speech-text pre-training methods have shown remarkable success in many speech and natural language processing tasks. How- ever, most previous pre-trained models are usu- ally tailored for one or two specific tasks, but fail to conquer a wide range of speech-text tasks. In addition, existing speech-text pre- training methods fail to explore the contextual information within a dialogue to enrich utter- ance representations. In this paper, we propose Speech-text dialog Pre-training for spoken dia- log understanding with ExpliCiT cRoss-Modal Alignment (SPECTRA), which is the first-ever speech-text dialog pre-training model. Con- cretely, to consider the temporality of speech modality, we design a novel temporal position prediction task to capture the speech-text align- ment. This pre-training task aims to predict the start and end time of each textual word in the corresponding speech waveform. In addition, to learn the characteristics of spoken dialogs, we generalize a response selection task from textual dialog pre-training to speech-text dialog pre-training scenarios. Experimental results on four different downstream speech-text tasks demonstrate the superiority of SPECTRA in learning speech-text alignment and multi-turn dialog context.1 1 Introduction In recent years, speech-text pre-training, which learns universal feature representations from a large training corpus (Chen et al., 2018; Li et al., 2021; Bapna et al., 2021), has achieved significant suc- cess in both uni-modal (Schneider et al., 2019; Dosovitskiy et al., 2020) and multi-modal (Lu et al., 2019; Radford et al., 2021) downstream tasks. Ex- ∗ Equal contribution. This work was conducted when Tianshu Yu and Haoyu Gao were interning at Alibaba. †† Min Yang and Yongbin Li are corresponding authors. 1For reproducibility, we release our code and pre- trained model at: https://github.com/AlibabaResearch/ DAMO-ConvAI/tree/main/SPECTRA. Today I am joined by my best friend for a gazillion years? My best friend Ruth here as my mental and emotional support. … Hey guys. Meet me. I've just give a little spiel about myself. My name's [MASK] like sounds like Bros. [MASK] MSA ERC SLU DST Downstream Tasks Pre-training Query Context Figure 1: An illustration of SPECTRA, which considers dialogue context and explicit alignment between text and speech during pre-training, and generalizes well on various downstream tasks. isting speech-text pre-training works mainly em- ployed multi-modal self-supervised pre-training ob- jectives, such as cross-modal masked data model- ing (Li et al., 2021; Kang et al., 2022a) and cross- modal contrastive learning (Sachidananda et al., 2022; Elizalde et al., 2022), which align the speech utterance representation to the corresponding text sentence representation. Despite the remarkable progress of previous speech-text pre-training models, there are …

Content truncated to 500 words