Effectively pretraining a speech translation decoder with Machine Translation data

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8014–8020, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8014 Effectively pretraining a speech translation decoder with Machine Translation data Ashkan Alinejad, Anoop Sarkar Simon Fraser University, Burnaby, BC, Canada {aalineja,anoop}@sfu.ca Abstract Directly translating from speech to text using an end-to-end approach is still challenging for many language pairs due to insufficient data. Although pretraining the encoder parameters using the Au- tomatic Speech Recognition (ASR) task improves the results in low resource settings, attempting to use pretrained parameters from the Neural Machine Translation (NMT) task has been largely unsuccess- ful in previous works. In this paper, we will show that by using an adversarial regularizer, we can bring the encoder representations of the ASR and NMT tasks closer even though they are in different modalities, and how this helps us effectively use a pretrained NMT decoder for speech translation. 1 Introduction Automatic Speech Translation (AST) aims to di- rectly translate audio signals in the source language into the text words in the target language. For many years, the pipeline of transcribing speech with ASR and then translating with the MT component was a standard method to address the speech translation problem. Having access to lots of data in many lan- guage pairs, the cascaded model for speech trans- lation can benefit from well-trained ASR and MT components and generate high-quality translations. In recent years, it has shown that we can re- move the transcription step and build an end-to-end model that is strong enough to compete with the cascaded model (Pino et al., 2019). Such models not only have lower inference latency, but they also do not suffer from the problem of errors that prop- agate from one component to the next. However, the scarcity of available resources is the main chal- lenge in this task, and a variety of methods are proposed to address this problem. One of the most effective approaches to increase the performance of AST systems is to pretrain the encoder using an ASR model (Bansal et al., 2018). While pretrain- ing the encoder by an ASR model even in different languages shows promising results (Bansal et al., 2019), using a pretrained MT decoder is not ben- eficial (Berard et al., 2018; Bansal et al., 2018) or slightly improve the result (Sperber et al., 2019) and even in some cases may worsen the results (Bahar et al., 2019). One explanation for this phenomenon is that the decoder works well only if its input comes from an encoder that it was trained with (Lample et al., 2018). To solve the problem of invariant encoder representations, we make use of an adversarial reg- ularizer in our loss function to bring the output of the ASR encoder closer to the input of MT decoder. We show that this modification can improve the BLEU score by +2.0 BLEU points. 2 Models 2.1 End-to-End Speech Translation Similar to conventional MT models, the speech translation task generates translated words …

Content truncated to 500 words