SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1663 - 1676 December 7-11, 2022 ©2022 Association for Computational Linguistics SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training Ziqiang Zhang1,∗, Long Zhou2,†, Junyi Ao3,∗, Shujie Liu2, Lirong Dai1, Jinyu Li2, Furu Wei2 1University of Science and Technology of China 2Microsoft 3The Chinese University of Hong Kong, Shenzhen Abstract The rapid development of single-modal pre- training has prompted researchers to pay more attention to cross-modal pre-training methods. In this paper, we propose a unified-modal speech-unit-text pre-training model, SpeechUT, to connect the representations of a speech encoder and a text decoder with a shared unit encoder. Leveraging hidden-unit as an interface to align speech and text, we can decompose the speech-to-text model into a speech-to-unit model and a unit-to-text model, which can be jointly pre-trained with unpaired speech and text data respectively. Our pro- posed SpeechUT is fine-tuned and evaluated on automatic speech recognition (ASR) and speech translation (ST) tasks. Experimental re- sults show that SpeechUT gets substantial im- provements over strong baselines, and achieves state-of-the-art performance on both the Lib- riSpeech ASR and MuST-C ST tasks. To better understand the proposed SpeechUT, detailed analyses are conducted. The code and pre- trained models are available at https://aka. ms/SpeechUT. 1 Introduction Self-supervised pre-training with large-scale unla- beled data obtains remarkable progress on various downstream tasks (Devlin et al., 2019; Radford et al., 2019; Dong et al., 2019; Baevski et al., 2020; Hsu et al., 2021; Chen et al., 2021). Specifically, pre-trained models, such as BERT (Devlin et al., 2019) and GPT (Radford et al., 2019), have exten- sively promoted the development of natural lan- guage processing (NLP). Researchers also develop many pre-trained speech models utilizing a mass of unlabeled audio data, e.g., wav2vec (Baevski et al., 2020) and HuBERT (Hsu et al., 2021). Al- though text and speech are two different modali- ties, they have a natural relationship because they ∗Work done during internship at Microsoft Research Asia. †Corresponding author (lozhou@microsoft.com). Speech Unit Unit Text Pre-train Fine-tune Shared Figure 1: A high-level illustration of SpeechUT. After pre-trained with speech-to-unit and unit-to-text tasks (blue arrows), the model with a shared unit encoder enables speech-to-text tasks for fine-tuning (red arrow). can be viewed as two kinds of expressions of lan- guage. Hence, joint pre-training of speech and text has received increasing attention from the research community in recent years (Ao et al., 2022a; Bapna et al., 2021; Tang et al., 2022). One line of speech-text joint pre-training builds a shared encoder to learn speech and text represen- tation jointly, such as SLAM (Bapna et al., 2021), which needs a random initialization of the de- coder parameter for fine-tuning an encoder-decoder model. Another line of studies, e.g., SpeechT5 (Ao et al., 2022a) and STPT (Tang et al., 2022), directly pre-trains an encoder-decoder model on speech and text corpus to boost the performance of automatic speech recognition (ASR) and speech translation (ST), leveraging unsupervised vector quantization (van den Oord …

Content truncated to 500 words