ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2819–2826 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2819 ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation Lifu Tu1 Richard Yuanzhe Pang2∗ Sam Wiseman1 Kevin Gimpel1 1Toyota Technological Institute at Chicago, Chicago, IL 60637, USA 2New York University, New York, NY 10011, USA {lifu,swiseman,kgimpel}@ttic.edu, yzpang@nyu.edu Abstract We propose to train a non-autoregressive ma- chine translation model to minimize the energy defined by a pretrained autoregressive model. In particular, we view our non-autoregressive translation system as an inference network (Tu and Gimpel, 2018) trained to minimize the autoregressive teacher energy. This con- trasts with the popular approach of training a non-autoregressive model on a distilled cor- pus consisting of the beam-searched outputs of such a teacher model. Our approach, which we call ENGINE (ENerGy-based In- ference NEtworks), achieves state-of-the-art non-autoregressive results on the IWSLT 2014 DE-EN and WMT 2016 RO-EN datasets, ap- proaching the performance of autoregressive models.1 1 Introduction The performance of non-autoregressive neural ma- chine translation (NAT) systems, which predict to- kens in the target language independently of each other conditioned on the source sentence, has been improving steadily in recent years (Lee et al., 2018; Ghazvininejad et al., 2019; Ma et al., 2019). One common ingredient in getting non-autoregressive systems to perform well is to train them on a corpus of distilled translations (Kim and Rush, 2016). This distilled corpus consists of source sentences paired with the translations produced by a pretrained au- toregressive “teacher” system. As an alternative to training non-autoregressive translation systems on distilled corpora, we instead propose to train them to minimize the energy de- fined by a pretrained autoregressive teacher model. That is, we view non-autoregressive machine trans- ∗Work partly done at Toyota Technological Institute at Chicago and the University of Chicago. 1Code is available at https://github.com/ lifu-tu/ENGINE lation systems as inference networks (Tu and Gim- pel, 2018, 2019; Tu et al., 2019) trained to mini- mize the teacher’s energy. This provides the non- autoregressive model with additional information related to the energy of the teacher, rather than just the approximate minimizers of the teacher’s energy appearing in a distilled corpus. In order to train inference networks to minimize an energy function, the energy must be differen- tiable with respect to the inference network out- put. We describe several approaches for relax- ing the autoregressive teacher’s energy to make it amenable to minimization with an inference network, and compare them empirically. We ex- periment with two non-autoregressive inference network architectures, one based on bidirectional RNNs and the other based on the transformer model of Ghazvininejad et al. (2019). In experiments on the IWSLT 2014 DE-EN and WMT 2016 RO-EN datasets, we show that train- ing to minimize the teacher’s energy significantly outperforms training with distilled outputs. Our approach, which we call ENGINE (ENerGy-based Inference NEtworks), achieves state-of-the-art re- sults for non-autoregressive translation on these datasets, approaching the results of the autoregres- sive teachers. Our hope is that ENGINE …

Content truncated to 500 words