FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4282–4292, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4282 FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow Xuezhe Ma⇤,1 Chunting Zhou⇤,1 Xian Li2 Graham Neubig1 Eduard Hovy1 1Language Technologies Institute, Carnegie Mellon University 2Facebook AI {xuezhem, chuntinz, gneubig, ehovy}@cs.cmu.edu xianl@fb.com Abstract Most sequence-to-sequence (seq2seq) models are autoregressive; they generate each token by conditioning on previously generated to- kens. In contrast, non-autoregressive seq2seq models generate all tokens in one pass, which leads to increased efficiency through parallel processing on hardware such as GPUs. How- ever, directly modeling the joint distribution of all tokens simultaneously is challenging, and even with increasingly complex model struc- tures accuracy lags significantly behind au- toregressive models. In this paper, we propose a simple, efficient, and effective model for non-autoregressive sequence generation using latent variable models. Specifically, we turn to generative flow, an elegant technique to model complex distributions using neural net- works, and design several layers of flow tai- lored for modeling the conditional density of sequential latent variables. We evaluate this model on three neural machine translation (NMT) benchmark datasets, achieving com- parable performance with state-of-the-art non- autoregressive NMT models and almost con- stant decoding time w.r.t the sequence length.1 1 Introduction Neural sequence-to-sequence (seq2seq) models (Bahdanau et al., 2015; Rush et al., 2015; Vinyals et al., 2015; Vaswani et al., 2017) generate an output sequence y = {y1, . . . , yT } given an in- put sequence x = {x1, . . . , xT 0} using condi- tional probabilities P✓(y|x) predicted by neural networks (parameterized by ✓). Most seq2seq models are autoregressive, mean- ing that they factorize the joint probability of the output sequence given the input sequence P✓(y|x) into the product of probabilities over the next to- ⇤ Equal contribution, in alphabetical order. 1https://github.com/XuezheMax/flowseq (a) (c) (b) Figure 1: (a) Autoregressive (b) non-autoregressive and (c) our proposed sequence generation models. x is the source, y is the target, and z are latent variables. ken in the sequence given the input sequence and previously generated tokens: P✓(y|x) = T Y t=1 P✓(yt|y<t, x). (1) Each factor, P✓(yt|y<t, x), can be implemented by function approximators such as RNNs (Bah- danau et al., 2015) and Transformers (Vaswani et al., 2017). This factorization takes the com- plicated problem of joint estimation over an ex- ponentially large output space of outputs y, and turns it into a sequence of tractable multi-class classification problems predicting yt given the pre- vious words, allowing for simple maximum log- likelihood training. However, this assumption of left-to-right factorization may be sub-optimal from a modeling perspective (Gu et al., 2019; Stern et al., 2019), and generation of outputs must be done through a linear left-to-right pass through the output tokens using beam search, which is not easily parallelizable on hardware such as GPUs. Recently, there has been work on non- autoregressive sequence generation for …

Content truncated to 500 words