Pre-training on high-resource speech recognition improves low-resource speech-to-text translation

Proceedings of NAACL-HLT 2019, pages 58–68 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 58 Pre-training on High-Resource Speech Recognition Improves Low-Resource Speech-to-Text Translation Sameer Bansal1 Herman Kamper2 Karen Livescu3 Adam Lopez1 Sharon Goldwater1 1School of Informatics, University of Edinburgh, UK 2E&E Engineering, Stellenbosch University, South Africa 3Toyota Technological Institute at Chicago, USA {sameer.bansal, sgwater, alopez}@inf.ed.ac.uk kamperh@sun.ac.za klivescu@ttic.edu Abstract We present a simple approach to improve di- rect speech-to-text translation (ST) when the source language is low-resource: we pre-train the model on a high-resource automatic speech recognition (ASR) task, and then fine-tune its parameters for ST. We demonstrate that our approach is effective by pre-training on 300 hours of English ASR data to improve Spanish- English ST from 10.8 to 20.2 BLEU when only 20 hours of Spanish-English ST train- ing data are available. Through an ablation study, we find that the pre-trained encoder (acoustic model) accounts for most of the im- provement, despite the fact that the shared lan- guage in these tasks is the target language text, not the source language audio. Apply- ing this insight, we show that pre-training on ASR helps ST even when the ASR language differs from both source and target ST lan- guages: pre-training on French ASR also im- proves Spanish-English ST. Finally, we show that the approach improves performance on a true low-resource task: pre-training on a com- bination of English ASR and French ASR im- proves Mboshi-French ST, where only 4 hours of data are available, from 3.5 to 7.1 BLEU. 1 Introduction Speech-to-text Translation (ST) has many potential applications for low-resource languages: for exam- ple in language documentation, where the source language is often unwritten or endangered (Be- sacier et al., 2006; Martin et al., 2015; Adams et al., 2016a,b; Anastasopoulos and Chiang, 2017); or in crisis relief, where emergency workers might need to respond to calls or requests in a foreign lan- guage (Munro, 2010). Traditional ST is a pipeline of automatic speech recognition (ASR) and ma- chine translation (MT), and thus requires tran- scribed source audio to train ASR and parallel text to train MT. These resources are often unavailable for low-resource languages, but for our potential applications, there may be some source language audio paired with target language text translations. In these scenarios, end-to-end ST is appealing. Recently, Weiss et al. (2017) showed that end- to-end ST can be very effective, achieving an im- pressive BLEU score of 47.3 on Spanish-English ST. But this result required over 150 hours of trans- lated audio for training, still a substantial resource requirement. By comparison, a similar system trained on only 20 hours of data for the same task achieved a BLEU score of 5.3 (Bansal et al., 2018). Other low-resource systems have similarly low accuracies (Anastasopoulos and Chiang, 2018; B´erard et al., 2018). To improve end-to-end ST in low-resource set- tings, we can try to leverage other data resources. For example, if we have transcribed audio in the source language, we can use multi-task …

Content truncated to 500 words