Bi-SimCut: A Simple Strategy for Boosting Neural Machine Translation

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3938 - 3948 July 10-15, 2022 ©2022 Association for Computational Linguistics Bi-SimCut: A Simple Strategy for Boosting Neural Machine Translation Pengzhi Gao, Zhongjun He, Hua Wu, and Haifeng Wang Baidu Inc. No. 10, Shangdi 10th Street, Beijing, 100085, China {gaopengzhi,hezhongjun,wu_hua,wanghaifeng}@baidu.com Abstract We introduce Bi-SimCut: a simple but effec- tive training strategy to boost neural machine translation (NMT) performance. It consists of two procedures: bidirectional pretraining and unidirectional finetuning. Both procedures uti- lize SimCut, a simple regularization method that forces the consistency between the output distributions of the original and the cutoff sen- tence pairs. Without leveraging extra dataset via back-translation or integrating large-scale pretrained model, Bi-SimCut achieves strong translation performance across five translation benchmarks (data sizes range from 160K to 20.2M): BLEU scores of 31.16 for en → de and 38.37 for de → en on the IWSLT14 dataset, 30.78 for en → de and 35.15 for de → en on the WMT14 dataset, and 27.17 for zh → en on the WMT17 dataset. Sim- Cut is not a new method, but a version of Cut- off (Shen et al., 2020) simplified and adapted for NMT, and it could be considered as a perturbation-based method. Given the univer- sality and simplicity of SimCut and Bi-SimCut, we believe they can serve as strong baselines for future NMT research. 1 Introduction The state of the art in machine translation has been dramatically improved over the past decade thanks to the neural machine translation (NMT) (Wu et al., 2016), and Transformer-based models (Vaswani et al., 2017) often deliver state-of-the-art (SOTA) translation performance with large-scale corpora (Ott et al., 2018). Along with the development in the NMT field, consistency training (Bachman et al., 2014) has been widely adopted and shown great promise to improve NMT performance. It simply regularizes the NMT model predictions to be invariant to either small perturbations applied to the inputs (Sato et al., 2019; Shen et al., 2020) and hidden states (Chen et al., 2021) or the model randomness and variance existed in the training procedure (Liang et al., 2021). Specifically, Shen et al. (2020) introduce a set of cutoff data augmentation methods and utilize Jensen-Shannon (JS) divergence loss to force the consistency between the output distributions of the original and the cutoff augmented samples in the training procedure. Despite its impressive perfor- mance, finding the proper values for the four addi- tional hyper-parameters introduced in cutoff aug- mentation seems to be tedious and time-consuming if there are limited resources available, which hin- ders its practical value in the NMT field. In this paper, our main goal is to provide a sim- ple, easy-to-reproduce, but tough-to-beat strategy for training NMT models. Inspired by cutoff aug- mentation (Shen et al., 2020) and virtual adversar- ial regularization (Sato et al., 2019) for NMT, we firstly introduce a simple yet effective regulariza- tion method named SimCut. Technically, SimCut is not a new method …

Content truncated to 500 words