Incorporating a Local Translation Mechanism into Non-autoregressive Translation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1067–1073, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1067 Incorporating a Local Translation Mechanism into Non-autoregressive Translation Xiang Kong∗, Zhisong Zhang∗, Eduard Hovy Language Technologies Institute, Carnegie Mellon University {xiangk,zhisongz,hovy}@cs.cmu.edu Abstract In this work, we introduce a novel local au- toregressive translation (LAT) mechanism into non-autoregressive translation (NAT) models so as to capture local dependencies among tar- get outputs. Specifically, for each target decod- ing position, instead of only one token, we pre- dict a short sequence of tokens in an autore- gressive way. We further design an efficient merging algorithm to align and merge the out- put pieces into one final output sequence. We integrate LAT into the conditional masked lan- guage model (CMLM; Ghazvininejad et al., 2019) and similarly adopt iterative decod- ing. Empirical results on five translation tasks show that compared with CMLM, our method achieves comparable or better performance with fewer decoding iterations, bringing a 2.5x speedup. Further analysis indicates that our method reduces repeated translations and per- forms better at longer sentences. The code for our model is available at https://github. com/shawnkx/NAT-with-Local-AT. 1 Introduction Traditional neural machine translation (NMT) mod- els (Sutskever et al., 2014; Cho et al., 2014; Bah- danau et al., 2014; Gehring et al., 2017; Vaswani et al., 2017) commonly make predictions in an incremental token-by-token way, which is called autoregressive translation (AT). Although this strat- egy can capture the full translation history, it has relatively high decoding latency. To make the de- coding more efficient, non-autoregressive transla- tion (NAT) (Gu et al., 2018) is introduced to gener- ate multiple tokens at once instead of one-by-one. However, with the conditional independence prop- erty (Gu et al., 2018), NAT models do not directly consider the dependencies among output tokens, which may cause errors of repeated translation and ∗ Zhisong and Xiang contributed equally for this paper Figure 1: An example of the LAT mechanism. For each decoding position, a short sequence of tokens is gener- ated in an autoregressive way. ⟨sop⟩ is the special start- of-piece symbol. ‘pos*’ denotes the hidden state from the decoder at that position. incomplete translation (Wang et al., 2019). There have been various methods in previous work (Stern et al., 2019; Gu et al., 2019; Ma et al., 2018; Wei et al., 2019; Ma et al., 2019; Tu et al., 2020) to miti- gate this problem, including iterative decoding (Lee et al., 2018; Ghazvininejad et al., 2019). In this work, we introduce a novel mechanism, i.e., local autoregressive translation (LAT), to take local target dependencies into consideration. For a decoding position, instead of generating one token, we predict a short sequence of tokens (which we call a translation piece) for the current and next few positions in an autoregressive way. A simple example is shown in Figure 1. With this mechanism, there can be overlapping tokens between nearby translation pieces. We take advantage of these redundancies, and apply a sim- ple algorithm to align …

Content truncated to 500 words