Like hiking? You probably enjoy nature: Persona-grounded Dialog with Commonsense Expansions

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1006–1015, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1006 Iterative Refinement in the Continuous Space for Non-Autoregressive Neural Machine Translation Jason Lee New York University jason@cs.nyu.edu Raphael Shu zomux2000@gmail.com Kyunghyun Cho New York University kyunghyun.cho@nyu.edu Abstract We propose an efficient inference procedure for non-autoregressive machine translation that iteratively refines translation purely in the continuous space. Given a continuous latent variable model for machine translation (Shu et al., 2020), we train an inference network to approximate the gradient of the marginal log probability of the target sentence, using only the latent variable as input. This allows us to use gradient-based optimization to find the target sentence at inference time that approxi- mately maximizes its marginal probability. As each refinement step only involves computa- tion in the latent space of low dimensional- ity (we use 8 in our experiments), we avoid computational overhead incurred by existing non-autoregressive inference procedures that often refine in token space. We compare our approach to a recently proposed EM-like in- ference procedure (Shu et al., 2020) that op- timizes in a hybrid space, consisting of both discrete and continuous variables. We evaluate our approach on WMT’14 En→De, WMT’16 Ro→En and IWSLT’16 De→En, and observe two advantages over the EM-like inference: (1) it is computationally efficient, i.e. each re- finement step is twice as fast, and (2) it is more effective, resulting in higher marginal proba- bilities and BLEU scores with the same num- ber of refinement steps. On WMT’14 En→De, for instance, our approach is able to decode 6.2 times faster than the autoregressive model with minimal degradation to translation quality (0.9 BLEU). 1 Introduction Most neural machine translation systems are autore- gressive, hence decoding latency grows linearly with respect to the length of the target sentence. For faster generation, several work proposed non- autoregressive models with sub-linear decoding latency given sufficient parallel computation (Gu et al., 2018a; Lee et al., 2018; Kaiser et al., 2018). As it is challenging to precisely model the depen- dencies among the tokens without autoregression, many existing non-autoregressive models first gen- erate an initial translation which is then iteratively refined to yield better output (Lee et al., 2018; Gu et al., 2019; Ghazvininejad et al., 2019). While vari- ous training objectives are used to admit refinement (e.g. denoising, evidence lowerbound maximiza- tion and mask language modeling), the generation process of these models is similar in that the re- finement process happens in the discrete space of sentences. Meanwhile, another line of work proposed to use continuous latent variables for non-autoregressive translation, such that the distribution of the tar- get sentences can be factorized over time given the latent variables (Ma et al., 2019; Shu et al., 2020). Unlike the models discussed above, finding the most likely target sentence under these mod- els requires searching over continuous latent vari- ables. To this end, Shu et al. (2020) proposed an EM-like inference procedure that optimizes over a hybrid space …

Content truncated to 500 words