On the Complementarity between Pre-Training and Random-Initialization for Resource-Rich Machine Translation

Proceedings of the 29th International Conference on Computational Linguistics, pages 5029–5034 October 12–17, 2022. 5029 On the Complementarity between Pre-Training and Random-Initialization for Resource-Rich Machine Translation Changtong Zan1∗, Liang Ding2†, Li Shen2, Yu Cao3, Weifeng Liu1†, Dacheng Tao2,3 1College of Control Science and Engineering, China University of Petroleum (East China) 2JD Explore Academy 3The University of Sydney b20050011@s.upc.edu.cn, dingliang1@jd.com Abstract Pre-Training (PT) of text representations has been successfully applied to low-resource Neu- ral Machine Translation (NMT). However, it usually fails to achieve notable gains (some- times, even worse) on resource-rich NMT on par with their Random-Initialization (RI) coun- terpart. We take the first step to investigate the complementarity between PT and RI in resource-rich scenarios via two probing anal- yses, and find that: 1 PT improves NOT the accuracy, but the generalization by achiev- ing flatter loss landscapes than that of RI; 2 PT improves NOT the confidence of lexical choice, but the negative diversity1 by assign- ing smoother lexical probability distributions than that of RI. Based on these insights, we pro- pose to combine their complementarities with a model fusion algorithm that utilizes optimal transport to align neurons between PT and RI. Experiments on two resource-rich translation benchmarks, WMT’17 English-Chinese (20M) and WMT’19 English-German (36M), show that PT and RI could be nicely complementary to each other, achieving substantial improve- ments considering both translation accuracy, generalization, and negative diversity. Prob- ing tools and code are released at: https:// github.com/zanchangtong/PTvsRI. 1 Introduction Pre-training (PT; Devlin et al., 2019; Liu et al., 2019 has achieved tremendous success in natural language processing fields. Inspired by BERT (De- vlin et al., 2019), recent works (Song et al., 2019; ∗ Work was done when Changtong was interning at JD Explore Academy. † Liang Ding and Weifeng Liu are the corresponding authors. 1Li et al. 2020 mentioned that the maximum likelihood estimation training treats all non-ground truth but semantic- relevant lexical predictions as being equally incorrect, poten- tially leading to the sharp lexical probability distribution. Here we use this term to refer to the higher probabilities assigned to the negative tokens. Lewis et al., 2020; Liu et al., 2020) attempt to lever- age sequence-to-sequence PT for neural machine translation (NMT; Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017) by leveraging a large amount of unlabeled (i.e. monolingual) sentences. While recent studies have empirically shown their benefit for the low-resource translation task where the labeled (i.e. parallel) sentences are lim- ited (Lewis et al., 2020; Song et al., 2019; Liu et al., 2020), we are generally confronted with resource- rich scenarios, e.g. millions of parallel sentence pairs, in WMT evaluations (Akhbardeh et al., 2021) and industries. For these resource-rich tasks, PT becomes less effective (sometimes, even worse) than their Random-Initialization (RI) counterparts, for example, as Zhu et al. 2020; Liu et al. 2020 re- ported, the PT underperforms RI if improperly uti- lized or significant amount of bi-text data is given. However, there is limited understanding of: 1 Why does PT fail …

Content truncated to 500 words