Harnessing Indirect Training Data for End-to-End Automatic Speech Translation: Tricks of the Trade

Harnessing Indirect Training Data for End-to-End Automatic Speech Translation: Tricks of the Trade Juan Pino1, Liezl Puzon1, Jiatao Gu1, Xutai Ma1,2, Arya D. McCarthy1,2, Deepak Gopinath1 1Facebook 2Johns Hopkins University {juancarabina,lie,jgu,dgopinath}@fb.com {xutai ma,arya}@jhu.edu Abstract For automatic speech translation (AST), end-to-end ap- proaches are outperformed by cascaded models that tran- scribe with automatic speech recognition (ASR), then trans- late with machine translation (MT). A major cause of the per- formance gap is that, while existing AST corpora are small, massive datasets exist for both the ASR and MT subsys- tems. In this work, we evaluate several data augmentation and pretraining approaches for AST, by comparing all on the same datasets. Simple data augmentation by translating ASR transcripts proves most effective on the English–French augmented LibriSpeech dataset, closing the performance gap from 8.2 to 1.4 BLEU, compared to a very strong cascade that could directly utilize copious ASR and MT data. The same end-to-end approach plus fine-tuning closes the gap on the English–Romanian MuST-C dataset from 6.7 to 3.7 BLEU. In addition to these results, we present practical rec- ommendations for augmentation and pretraining approaches. Finally, we decrease the performance gap to 0.01 BLEU us- ing a Transformer-based architecture. 1. Introduction Automatic speech-to-text translation (AST) is the task of transforming speech input to its corresponding textual trans- lation. Traditionally, AST has been conducted via a cascade approach [1, 2]: an automatic speech recognition (ASR) sys- tem creates a transcript of the speech signal, which is then translated by a machine translation (MT) system. Recent ap- proaches use end-to-end neural network models [3, 4], which are directly inspired by end-to-end models for ASR [5, 6]. Cascade approaches can benefit from large amounts of train- ing data available to its components for certain language pairs. For example, an English ASR model can be trained on 960 hours of speech [7], and an English–French translation model can be trained on about 40 million sentence pairs [8]. End-to-end approaches have very limited data avail- able [9, 1, 10]; nevertheless, they present several benefits. First, end-to-end models can enable lower inference latency since they involve only one prediction. Second, it may be easier to reduce the model size for a single integrated model. Finally, end-to-end approaches avoid compounding errors Synthetic English audio English text French text English audio English text French text English audio English text Synthetic French text MT model trained on En–Fr External TTS model End-to-end source End-to-end target MT data AST data ASR data Figure 1: AST datasets typically include three parts: recorded speech, transcripts, and translations. While cas- caded models can leverage these, an advantage over end- to-end systems is that they can also leverage data that pro- vide adjacent pairs—which are far more prevalent. The ap- proaches we investigate involve completing the triplets, so end-to-end systems can also benefit from incomplete triplets. from the ASR and MT models. End-to-end models for AST have been shown to perform better than or on par with cascade models [4, 11] when both are trained only …

Content truncated to 500 words