Syntactically Supervised Transformers for Faster Neural Machine Translation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1269–1281 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1269 Syntactically Supervised Transformers for Faster Neural Machine Translation Nader Akoury, Kalpesh Krishna, Mohit Iyyer College of Information and Computer Sciences University of Massachusetts Amherst {nsa,kalpesh,miyyer}@cs.umass.edu Abstract Standard decoders for neural machine trans- lation autoregressively generate a single tar- get token per time step, which slows infer- ence especially for long outputs. While ar- chitectural advances such as the Transformer fully parallelize the decoder computations at training time, inference still proceeds sequen- tially. Recent developments in non- and semi- autoregressive decoding produce multiple to- kens per time step independently of the oth- ers, which improves inference speed but deteri- orates translation quality. In this work, we pro- pose the syntactically supervised Transformer (SynST), which first autoregressively predicts a chunked parse tree before generating all of the target tokens in one shot conditioned on the predicted parse. A series of controlled experi- ments demonstrates that SynST decodes sen- tences ∼ 5× faster than the baseline autore- gressive Transformer while achieving higher BLEU scores than most competing methods on En-De and En-Fr datasets. 1 Introduction Most models for neural machine translation (NMT) rely on autoregressive decoders, which predict each token ti in the target language one by one condi- tioned on all previously-generated target tokens t1···i−1 and the source sentence s. For downstream applications of NMT that prioritize low latency (e.g., real-time translation), autoregressive decod- ing proves expensive, as decoding time in state- of-the-art attentional models such as the Trans- former (Vaswani et al., 2017) scales quadratically with the number of target tokens. In order to speed up test-time translation, non- autoregressive decoding methods produce all tar- get tokens at once independently of each other (Gu et al., 2018; Lee et al., 2018), while semi- autoregressive decoding (Wang et al., 2018; Stern Cats > sleep > a > lot Katzen schlafen viel Cats sleep a lot Cats sleep > a lot > > > Cats sleep a lot NP1 > VP3 > Cats sleep a lot autoregressive non-autoregressive semi-autoregressive Latent Transformer SynST (ours) Figure 1: Comparison of different methods designed to increase decoding speed. The arrow > indicates the be- ginning of a new decode step conditioned on everything that came previously. The latent Transformer produces a sequence of discrete latent variables, whereas SynST produces a sequence of syntactic constituent identifiers. et al., 2018) trades off speed for quality by reduc- ing (but not completely eliminating) the number of sequential computations in the decoder (Figure 1). We choose the latent Transformer (LT) of Kaiser et al. (2018) as a starting point, which merges both of these approaches by autoregressively generating a short sequence of discrete latent variables before non-autoregressively producing all target tokens conditioned on the generated latent sequence. Kaiser et al. (2018) experiment with increas- ingly complex ways of learning their discrete latent space, some of which obtain small BLEU improve- ments over a …

Content truncated to 500 words