Improving Neural Machine Translation by Bidirectional Training

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3278–3284 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3278 Improving Neural Machine Translation by Bidirectional Training Liang Ding The University of Sydney ldin3097@sydney.edu.au Di Wu Peking University inbath@163.com Dacheng Tao JD Explore Academy, JD.com dacheng.tao@gmail.com Abstract We present a simple and effective pretrain- ing strategy – bidirectional training (BiT) for neural machine translation. Specifically, we bidirectionally update the model parameters at the early stage and then tune the model nor- mally. To achieve bidirectional updating, we simply reconstruct the training samples from “src→tgt” to “src+tgt→tgt+src” without any complicated model modifications. Notably, our approach does not increase any parameters or training steps, requiring the parallel data merely. Experimental results show that BiT pushes the SOTA neural machine translation performance across 15 translation tasks on 8 language pairs (data sizes range from 160K to 38M) significantly higher. Encouragingly, our proposed model can complement existing data manipulation strategies, i.e. back translation, data distillation and data diversification. Ex- tensive analyses show that our approach func- tions as a novel bilingual code-switcher, ob- taining better bilingual alignment. 1 Introduction Recent years have seen a surge of interest in neural machine translation (NMT, Luong et al., 2015; Wu et al., 2016; Gehring et al., 2017; Vaswani et al., 2017) where it benefits from a massive amount of training data. But obtaining such large amounts of parallel data is not-trivial in most machine transla- tion scenarios. For example, there are many low- resource language pairs (e.g. English-to-Tamil), which lack adequate parallel data for training. Although many approaches about fully exploit- ing the parallel and monolingual data are pro- posed, e.g. back translation (Sennrich et al., 2016a), knowledge distillation (Kim and Rush, 2016) and data diversification (Nguyen et al., 2020), the pre- requisite of these approaches is to build a well- performed baseline model based on the parallel data. However, Koehn and Knowles (2017); Lam- ple et al. (2018); Sennrich and Zhang (2019) em- pirically reveal that NMT runs worse than their statistical or even unsupervised counterparts in low- resource conditions. Here naturally arise a ques- tion: Can we find a strategy to consistently improve NMT performance given the parallel data merely? We decide to find a solution from human learn- ing behavior. Pavlenko and Jarvis (2002); Dworin (2003); Chen et al. (2015) show that bidirec- tional language learning helps master bilingual- ism. In the context of machine translation, both the source→target and target→source language mappings may benefit bilingual modeling, which motivates many recent studies, e.g. dual learn- ing (He et al., 2016) and symmetric training (Cohn et al., 2016; Liang et al., 2007). However, their approaches rely on external resources (e.g. word alignment or monolingual data) or complicated model modifications, which limit the applicabil- ity of the method to a broader range of languages and model structures. Accordingly, we turn to pro- pose a simple data manipulation strategy and trans- fer the bidirectional relationship through bidirec- tional training (§2.2). The core idea is …

Content truncated to 500 words