Unified Speech-Text Pre-training for Speech Translation and Recognition

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1488 - 1499 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Unified Speech-Text Pre-training for Speech Translation and Recognition Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, Juan Pino Meta AI {yuntang,hygong,dnn,changhan,wnhsu,jgu,abaevski,xianl, abdo,michaelauli,juancarabina}@fb.com Abstract We describe a method to jointly pre-train speech and text in an encoder-decoder mod- eling framework for speech translation and recognition. The proposed method incor- porates four self-supervised and supervised subtasks for cross modality learning. A self-supervised speech subtask leverages un- labelled speech data, and a (self-)supervised text to text subtask makes use of abundant text training data. Two auxiliary supervised speech tasks are included to unify speech and text modeling space. Our contribution lies in in- tegrating linguistic information from the text corpus into the speech pre-training. Detailed analysis reveals learning interference among subtasks. Two pre-training configurations for speech translation and recognition, respec- tively, are presented to alleviate subtask inter- ference. Our experiments show the proposed method can effectively fuse speech and text in- formation into one model. It achieves between 1.7 and 2.3 BLEU improvement above the state of the art on the MUST-C speech transla- tion dataset and comparable WERs to wav2vec 2.0 on the LIBRISPEECH speech recognition task. 1 1 Introduction Pre-training can learn universal feature represen- tations from a large training corpus and is benefi- cial for downstream tasks with limited amounts of training data (Peters et al., 2018; van den Oord et al., 2018; Chung et al., 2018; Zoph et al., 2020). With the advancement of computational power and self-supervised pre-training approaches, large volumes of unlabeled data may now be used in pre-training. Methods, such as BERT (Devlin et al., 2019), BART (Lewis et al., 2020b) and wav2vec2.0 (Baevski et al., 2020b), have emerged as the backbone of many speech and natural lan- guage processing tasks. 1https://github.com/pytorch/fairseq/tree/main/ examples/speech text joint to text. The aforementioned pre-training methods focus on learning feature representation either from text or speech. Many speech applications combine in- formation learnt from both speech and text corpora to achieve state of the art results. In speech process- ing, transcribed speech training data is generally very scarce for many languages. It is difficult to build robust linguistic knowledge representation solely based on labeled speech training data. Jia et al. (2019); Chen et al. (2021) propose to gen- erate synthetic data from text to augment speech training corpus. Li et al. (2021) demonstrate that models initialized with pre-trained wav2vec2.0 and mBART (Liu et al., 2020) modules are competi- tive for the multilingual speech to text translation task. Chuang et al. (2020) propose to concatenate the acoustic model and BERT model for speech Q&A. Chung et al. (2021b) align speech utterance representation to the corresponding text sentence representation, in which both representations are generated from unsupervised pre-trained models, for speech understanding. In this study, we are interested in pre-training for speech to …

Content truncated to 500 words