Fast Interleaved Bidirectional Sequence Generation

Proceedings of the 5th Conference on Machine Translation (WMT), pages 503–515 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 503 Fast Interleaved Bidirectional Sequence Generation Biao Zhang1 Ivan Titov1,2 Rico Sennrich3,1 1School of Informatics, University of Edinburgh 2ILLC, University of Amsterdam 3Department of Computational Linguistics, University of Zurich B.Zhang@ed.ac.uk, ititov@inf.ed.ac.uk, sennrich@cl.uzh.ch Abstract Independence assumptions during sequence generation can speed up inference, but paral- lel generation of highly inter-dependent tokens comes at a cost in quality. Instead of assuming independence between neighbouring tokens (semi-autoregressive decoding, SA), we take inspiration from bidirectional sequence gener- ation and introduce a decoder that generates target words from the left-to-right and right-to- left directions simultaneously. We show that we can easily convert a standard architecture for unidirectional decoding into a bidirectional decoder by simply interleaving the two direc- tions and adapting the word positions and self- attention masks. Our interleaved bidirectional decoder (IBDecoder) retains the model sim- plicity and training efficiency of the standard Transformer, and on five machine translation tasks and two document summarization tasks, achieves a decoding speedup of ∼2× com- pared to autoregressive decoding with compa- rable quality. Notably, it outperforms left-to- right SA because the independence assump- tions in IBDecoder are more felicitous. To achieve even higher speedups, we explore hy- brid models where we either simultaneously predict multiple neighbouring tokens per direc- tion, or perform multi-directional decoding by partitioning the target sequence. These meth- ods achieve speedups to 4×–11× across dif- ferent tasks at the cost of <1 BLEU or <0.5 ROUGE (on average).1 1 Introduction Neural sequence generation aided by encoder- decoder models (Bahdanau et al., 2015; Vaswani et al., 2017) has achieved great success in recent years (Bojar et al., 2018; Song et al., 2019; Raf- fel et al., 2019; Karita et al., 2019), but still suf- fers from slow inference. One crucial bottleneck 1Source code is released at https://github.com/ bzhangGo/zero. lies in its generative paradigm which factorizes the conditional probability along the target sequence y = {y1, y2, . . . , yn} of length n as follows: p(y|x) = n � t=1 p (yt|y<t, x) , (1) where x = {x1, x2, . . . , xm} is the source se- quence of length m. This factorization determines that target words can only be generated one-by-one in a sequential and unidirectional manner, which limits the decoding efficiency. A promising direction to break this barrier is to generate multiple target words at one decoding step to improve the parallelization of inference (Gu et al., 2018; Stern et al., 2018). However, this intro- duces independence assumptions that hurt transla- tion quality, since words produced in parallel are in fact likely to be inter-dependent. We hypothesize that there are groups of words that are less likely to be strongly inter-dependent than neighbouring words, which will allow for better parallelization. Inspired by bidirectional modeling (Zhang et al., 2019b, 2020), we resort to an alternative probabilis- tic factorization: pBD(y|x) = ⌈n/2⌉ � t=1 p �−→ yt, ←− yt′|−→ y<t, ←−− y>t′, …

Content truncated to 500 words