Bi-Directional Differentiable Input Reconstruction for Low-Resource Neural Machine Translation

Proceedings of NAACL-HLT 2019, pages 442–448 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 442 Bi-Directional Differentiable Input Reconstruction for Low-Resource Neural Machine Translation Xing Niu University of Maryland xingniu@cs.umd.edu Weijia Xu University of Maryland weijia@cs.umd.edu Marine Carpuat University of Maryland marine@cs.umd.edu Abstract We aim to better exploit the limited amounts of parallel text available in low-resource set- tings by introducing a differentiable recon- struction loss for neural machine translation (NMT). This loss compares original inputs to reconstructed inputs, obtained by back- translating translation hypotheses into the in- put language. We leverage differentiable sam- pling and bi-directional NMT to train models end-to-end, without introducing additional pa- rameters. This approach achieves small but consistent BLEU improvements on four lan- guage pairs in both translation directions, and outperforms an alternative differentiable re- construction strategy based on hidden states. 1 Introduction Neural Machine Translation (NMT) performance degrades sharply when parallel training data is limited (Koehn and Knowles, 2017). Past work has addressed this problem by leveraging mono- lingual data (Sennrich et al., 2016a; Ramachan- dran et al., 2017) or multilingual parallel data (Zoph et al., 2016; Johnson et al., 2017; Gu et al., 2018a). We hypothesize that the traditional train- ing can be complemented by better leveraging lim- ited training data. To this end, we propose a new training objective for this model by augmenting the standard translation cross-entropy loss with a differentiable input reconstruction loss to fur- ther exploit the source side of parallel samples. Input reconstruction is motivated by the idea of round-trip translation. Suppose sentence f is translated forward to e using model θfe and then translated back to ˆf using model θef, then e is more likely to be a good translation if the distance between ˆf and f is small (Brislin, 1970). Prior work applied round-trip translation to monolin- gual examples and sampled the intermediate trans- lation e from a K-best list generated by model θfe using beam search (Cheng et al., 2016; He et al., 2016). However, beam search is not differ- entiable which prevents back-propagating recon- struction errors to θfe. As a result, reinforcement learning algorithms, or independent updates to θfe and θef were required. In this paper, we focus on the problem of mak- ing input reconstruction differentiable to simplify training. In past work, Tu et al. (2017) addressed this issue by reconstructing source sentences from the decoder’s hidden states. However, this re- construction task can be artificially easy if hid- den states over-memorize the input. This approach also requires a separate auxiliary reconstructor, which introduces additional parameters. We propose instead to combine benefits from differentiable sampling and bi-directional NMT to obtain a compact model that can be trained end- to-end with back-propagation. Specifically, • Translations are sampled using the Straight- Through Gumbel Softmax (STGS) estima- tor (Jang et al., 2017; Bengio et al., 2013), which allows back-propagating reconstruc- tion errors. • Our approach builds on the bi-directional NMT model (Niu et al., 2018; Johnson et al., 2017), which improves …

Content truncated to 500 words