Non-Autoregressive Machine Translation with Latent Alignments

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1098–1108, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1098 Non-Autoregressive Machine Translation with Latent Alignments Chitwan Saharia∗† Google Research, Brain Team sahariac@google.com William Chan∗ Google Research, Brain Team williamchan@google.com Saurabh Saxena Google Research, Brain Team srbs@google.com Mohammad Norouzi Google Research, Brain Team mnorouzi@google.com Abstract This paper presents two strong methods, CTC and Imputer, for non-autoregressive machine translation that model latent alignments with dynamic programming. We revisit CTC for machine translation and demonstrate that a simple CTC model can achieve state-of-the- art for single-step non-autoregressive machine translation, contrary to what prior work in- dicates. In addition, we adapt the Imputer model for non-autoregressive machine transla- tion and demonstrate that Imputer with just 4 generation steps can match the performance of an autoregressive Transformer baseline. Our latent alignment models are simpler than many existing non-autoregressive translation base- lines; for example, we do not require target length prediction or re-scoring with an autore- gressive model. On the competitive WMT’14 En→De task, our CTC model achieves 25.7 BLEU with a single generation step, while Im- puter achieves 27.5 BLEU with 2 generation steps, and 28.0 BLEU with 4 generation steps. This compares favourably to the autoregres- sive Transformer baseline at 27.8 BLEU. 1 Introduction Non-autoregressive neural machine translation (Gu et al., 2018) aims to enable the parallel gen- eration of output tokens without sacrificing trans- lation quality. There has been a surge of re- cent interest in this family of efficient decoding models, resulting in the development of iterative refinement (Lee et al., 2018), CTC models (Li- bovicky and Helcl, 2018), insertion-based meth- ods (Stern et al., 2019; Chan et al., 2019b), edit- based methods (Gu et al., 2019; Ruis et al., 2019), masked language models (Ghazvininejad et al., 2019, 2020b), and normalizing flow models (Ma et al., 2019). Some of these methods generate the ∗Equal contribution. †Work done as part of the Google AI Residency. output tokens in a constant number of steps (Gu et al., 2018; Libovicky and Helcl, 2018; Lee et al., 2018; Ghazvininejad et al., 2019, 2020b), while others require a logarithmic number of generation steps (Stern et al., 2019; Chan et al., 2019b,a; Li and Chan, 2019). Recent progress has decreased the gap between autoregressive and non-autoregressive models’ translation scores. However, non-autoregressive models often suffer from two main limitations: 1. First, most non-autoregressive models assume that the output tokens are conditionally inde- pendent given the input. This leads to the weak- ness of such models in generating multi-modal outputs (Gu et al., 2018), and materializes in the form of token repetitions in the decoded out- puts. Addressing this limitation generally in- volves stochastic search algorithms like noisy parallel decoding (Gu et al., 2018), iterative de- coding (Ghazvininejad et al., 2019, 2020b), or simple but less effective heuristic methods such as collapsing repetitions (Lee et al., 2018). 2. The second limitation of many prior non- autoregressive models is the requirement of output length prediction as a …

Content truncated to 500 words