PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 800 - 810 July 10-15, 2022 ©2022 Association for Computational Linguistics PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining Machel Reid The University of Tokyo machelreid@weblab.t.u-tokyo.ac.jp Mikel Artetxe Meta AI artetxe@fb.com Abstract Despite the success of multilingual sequence- to-sequence pretraining, most existing ap- proaches rely on monolingual corpora, and do not make use of the strong cross-lingual sig- nal contained in parallel data. In this paper, we present PARADISE (PARAllel & Denoising Integration in SEquence-to-sequence mod- els), which extends the conventional denoising objective used to train these models by (i) re- placing words in the noised sequence accord- ing to a multilingual dictionary, and (ii) pre- dicting the reference translation according to a parallel corpus instead of recovering the orig- inal sequence. Our experiments on machine translation and cross-lingual natural language inference show an average improvement of 2.0 BLEU points and 6.7 accuracy points from in- tegrating parallel data into pretraining, respec- tively, obtaining results that are competitive with several popular models at a fraction of their computational cost.1 1 Introduction Multilingual sequence-to-sequence pretraining has achieved strong results both in cross-lingual clas- sification (Xue et al., 2021) and machine transla- tion (Liu et al., 2020). These models are usually pretrained on combined monolingual corpora in multiple languages using some form of denoising objective. More concretely, they noise each se- quence x with a noising function gφ, and maximize the probability of recovering x given gφ(x): ℓmono(x) = − log P � x|gφ(x) � (1) Common noising functions include sentence- permutation and span masking (Lewis et al., 2020; Liu et al., 2020). While these methods obtain strong cross-lingual performance without parallel data, they are usually 1Source code available at https://github.com/ machelreid/paradise trained at a scale that is prohibitive for most NLP practitioners. At the same time, it has been argued that the strict unsupervised scenario is not realis- tic (Artetxe et al., 2020), and parallel data could provide a stronger signal and make training more efficient. Motivated by this, we propose PARADISE, a pre- training method for sequence-to-sequence models that exploits both word-level and sentence-level parallel data. The core idea of our approach is to augment the conventional denoising objective intro- duced above by (i) replacing words in the noised se- quence according to a bilingual dictionary, and (ii) predicting the reference translation rather than the input sequence. Despite their simplicity, we find that both techniques bring substantial gains over conventional pretraining on monolingual data, as evaluated both in machine translation and zero-shot cross-lingual transfer. Our results are competitive with several popular models despite using only a fraction of the compute, providing strong support for the importance of the inclusion of parallel in- formation in smaller-scale multilingual pretraining methods. 2 Proposed method As illustrated in Figure 1, we propose two meth- ods for introducing parallel data into pretraining: dictionary denoising and bitext denoising. Dictionary denoising. Our first method encour- ages learning …

Content truncated to 500 words