SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5723 - 5738 May 22-27, 2022 c⃝2022 Association for Computational Linguistics SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing Junyi Ao1,2,∗, Rui Wang3,∗, Long Zhou4,∗, Chengyi Wang4, Shuo Ren4, Yu Wu4, Shujie Liu4, Tom Ko1, Qing Li2, Yu Zhang1,5, Zhihua Wei3, Yao Qian4, Jinyu Li4, Furu Wei4 1Department of Computer Science and Engineering, Southern University of Science and Technology 2Department of Computing, The Hong Kong Polytechnic University 3Department of Computer Science and Technology, Tongji University 4Microsoft 5Peng Cheng Laboratory Abstract Motivated by the success of T5 (Text-To- Text Transfer Transformer) in pre-trained nat- ural language processing models, we pro- pose a unified-modal SpeechT5 framework that explores the encoder-decoder pre-training for self-supervised speech/text representation learning. The SpeechT5 framework consists of a shared encoder-decoder network and six modal-specific (speech/text) pre/post-nets. Af- ter preprocessing the input speech/text through the pre-nets, the shared encoder-decoder net- work models the sequence-to-sequence trans- formation, and then the post-nets generate the output in the speech/text modality based on the output of the decoder. Leveraging large-scale unlabeled speech and text data, we pre-train SpeechT5 to learn a unified-modal representa- tion, hoping to improve the modeling capabil- ity for both speech and text. To align the tex- tual and speech information into this unified semantic space, we propose a cross-modal vec- tor quantization approach that randomly mixes up speech/text states with latent units as the interface between encoder and decoder. Ex- tensive evaluations show the superiority of the proposed SpeechT5 framework on a wide va- riety of spoken language processing tasks, in- cluding automatic speech recognition, speech synthesis, speech translation, voice conver- sion, speech enhancement, and speaker iden- tification. We release our code and model at https://github.com/microsoft/ SpeechT5. 1 Introduction Starting with ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019), substantial work has shown that pre-trained models can significantly improve in various natural language processing (NLP) tasks ∗Equal contribution. Work is done by the first two authors during internship at Microsoft Research Asia. Correspondence to: Long Zhou (lozhou@microsoft.com) Figure 1: An illustration of the SpeechT5 frame- work, which treats spoken language processing tasks as a speech/text to speech/text format, including au- tomatic speech recognition (ASR), speech translation (ST), speech identification (SID), text to speech (TTS), voice conversion (VC), and speech enhancement (SE). (Radford et al., 2019; CONNEAU and Lample, 2019; Yang et al., 2019; Dong et al., 2019; Lewis et al., 2020). Following the pre-training techniques in NLP, self-supervised speech representation learn- ing has also been investigated and shown promising results, benefiting from richly learned representa- tions (Chung and Glass, 2018; Chuang et al., 2020; Song et al., 2019; Baevski et al., 2020; Wang et al., 2021; Hsu et al., 2021; Chung et al., 2021a), such as wav2vec 2.0 (Baevski et al., 2020) and HuBERT (Hsu et al., 2021). However, previous speech pre-training work suf- fers from two problems: (1) most of them learn the speech representation with only unlabeled …

Content truncated to 500 words