Dual Reconstruction: a Unifying Objective for Semi-Supervised Neural Machine Translation

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2006–2020 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2006 Dual Reconstruction: a Unifying Objective for Semi-Supervised Neural Machine Translation Weijia Xu University of Maryland weijia@cs.umd.edu Xing Niu∗ Amazon AI xingniu@amazon.com Marine Carpuat University of Maryland marine@cs.umd.edu Abstract While Iterative Back-Translation and Dual Learning effectively incorporate monolingual training data in neural machine translation, they use different objectives and heuristic gradient approximation strategies, and have not been extensively compared. We intro- duce a novel dual reconstruction objective that provides a unified view of Iterative Back- Translation and Dual Learning. It motivates a theoretical analysis and controlled empirical study on German-English and Turkish-English tasks, which both suggest that Iterative Back- Translation is more effective than Dual Learn- ing despite its relative simplicity. 1 Introduction Taking advantage of monolingual training data via Back-Translation (Sennrich et al., 2016a), Iterative Back-Translation (Zhang et al., 2018; Cotterell and Kreutzer, 2018) or Dual Learning (He et al., 2016) has become a de facto requirement for building high quality Neural Machine Translation (NMT) systems (Edunov et al., 2018; Hassan et al., 2018). However, these methods rely on unrelated heuristic optimization objectives, and it is not clear what their respective strengths and weaknesses are, nor how they relate to the ideal but intractable objec- tive of maximizing the marginal likelihood of the monolingual data (i.e., pθ(y) = � x pθ(y | x)q(x) given target sentences y, an NMT model pθ(y | x), and the prior distribution q(x) on source x). Instead of proposing new methods, this paper sheds new light on how these established tech- niques work and how to use them. We introduce a dual reconstruction objective to theoretically ground the comparison of semi-supervised train- ing strategies that leverage monolingual data from both source and target languages (Figure 1). In Sec- tion 3, we show that, under some assumptions, this ∗Work was done at the University of Maryland. NMT Model 𝑝" 𝑦$ $𝑥) NMT Model 𝑞( 𝑥$ $𝑦) $𝑦) $𝑥 $$𝐽+ = Ε.)~01 .$ $2)𝑞( 𝑥$ $𝑦)) NMT Model 𝑞( 𝑥$ $𝑦) NMT Model 𝑝" 𝑦$ $𝑥) $𝑥) $𝑦 source-target-source reconstruction target-source-target reconstruction inference reconstruction $$𝐽3 = Ε2)~45 2$ $.)𝑝" 𝑦$ $𝑥)) Figure 1: Our dual reconstruction objective sums 1) a target-source-target objective J1 on target sen- tences y using the NMT model qφ(x | y) for infer- ence and pθ(y | x) for reconstruction, and 2) a source- target-source objective J2 on source sentences x using pθ(y | x) for inference and qφ(x | y) for reconstruction. Models connected by dotted arrows share parameters. objective remarkably shares the same global opti- mum as the intractable marginal likelihood objec- tive where the model’s marginal distribution pθ(y) coincides with the target sentence distribution p(y). We also show that Iterative Back-Translation (IBT) and Dual Learning can be viewed as different ways to approximate its optimization. Theory suggests that IBT approximates the dual reconstruction objective more closely than the more complex Dual Learning approach, and in particular that Dual …

Content truncated to 500 words