A Comparison between Pre-training and Large-scale Back-translation for Neural Machine Translation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1718–1732 August 1–6, 2021. ©2021 Association for Computational Linguistics 1718 A Comparison between Pre-training and Large-scale Back-translation for Neural Machine Translation Dandan Huang1,2∗, Kun Wang3∗, Yue Zhang1,2† 1 School of Engineering, Westlake University 2 Institute of Advanced Technology, Westlake Institute for Advanced Study 3 Microsoft STCA NLP Group {huangdandan,zhangyue}@westlake.edu.cn, wangkun@microsoft.com Abstract BERT has been studied as a promising tech- nique to improve NMT. Given that BERT is based on the similar Transformer architecture to NMT and the current datasets for most MT tasks are rather large, how pre-training has managed to outperform standard Transformer NMT models is underestimated. We compare MT engines trained with pre-trained BERT and back-translation with incrementally larger amounts of data, implementing the two most widely-used monolingual paradigms. We ana- lyze their strengths and weaknesses based on both standard automatic metrics and intrinsic test suites that comprise a large range of lin- guistic phenomena. Primarily, we find that 1) BERT has limited advantages compared with large-scale back-translation in accuracy and consistency on morphology and syntax; 2) BERT can boost the Transformer baseline in semantic and pragmatic tasks which involve in- tensive understanding; 3) pre-training on huge datasets may introduce inductive social bias thus affects translation fairness. 1 Introduction Neural machine translation (NMT) has shown promising results as an end-to-end approach to automatic translation (Sutskever et al., 2014; Bah- danau et al., 2014; Vaswani et al., 2017). One reason for its success is the availability of large amounts of training resources such as parallel cor- pora with high quality. For low-resource languages or domain-specific settings, monolingual data have also been effectively used by NMT systems (Zhang and Zong, 2016; Siddhant et al., 2020), providing rich linguistic features for translation. Two lines of work have been done on leveraging monolingual corpora to improve translation qual- ity. One approach is back-translation (Bojar and ∗ Equal contribution. † Corresponding author. Tamchyna, 2011; Sennrich et al., 2016), in which an auxiliary target-to-source system is trained on genuine bitext, and then used to generate synthetic text from a large monolingual corpus on the target side. The synthetic and genuine pairs are then used together to train a source-to-target MT model. An alternative method of using monolingual data is the pre-trained language model (Devlin et al., 2019; Radford et al., 2019), a neural net- work trained over large texts and can be incorpo- rated into standard NMT encoder-decoder archi- tectures (Jean et al., 2015; Gulcehre et al., 2015; Zhu et al., 2020). Pre-trained language models have led to improvements in NMT results across low-resource scenarios (Song et al., 2019), cross- lingual transfers (Conneau and Lample, 2019; Liu et al., 2020) and code-switching settings (Yang et al., 2020). Among these two dominant monolingual paradigms, there has been relatively more work investigating how back-translation helps NMT. For example, initial studies show that back-translation is beneficial to machine translation by producing more fluent outputs (Edunov et al., 2020). How- ever, relatively little work has focused on how …

Content truncated to 500 words