Mask-Predict: Parallel Decoding of Conditional Masked Language Models

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 6112–6121, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 6112 Mask-Predict: Parallel Decoding of Conditional Masked Language Models Marjan Ghazvininejad∗ Omer Levy∗ Yinhan Liu∗ Luke Zettlemoyer Facebook AI Research Seattle, WA Abstract Most machine translation systems generate text autoregressively from left to right. We, instead, use a masked language modeling ob- jective to train a model to predict any subset of the target words, conditioned on both the input text and a partially masked target trans- lation. This approach allows for efficient it- erative decoding, where we first predict all of the target words non-autoregressively, and then repeatedly mask out and regenerate the subset of words that the model is least confi- dent about. By applying this strategy for a con- stant number of iterations, our model improves state-of-the-art performance levels for non- autoregressive and parallel decoding transla- tion models by over 4 BLEU on average. It is also able to reach within about 1 BLEU point of a typical left-to-right transformer model, while decoding significantly faster.1 1 Introduction Most machine translation systems use sequen- tial decoding strategies where words are predicted one-by-one. In this paper, we present a model and a parallel decoding algorithm which, for a rela- tively small sacrifice in performance, can be used to generate translations in a constant number of decoding iterations. We introduce conditional masked language models (CMLMs), which are encoder-decoder ar- chitectures trained with a masked language model objective (Devlin et al., 2018; Lample and Con- neau, 2019). This change allows the model to learn to predict, in parallel, any arbitrary subset of masked words in the target translation. We use transformer CMLMs, where the decoder’s self at- tention (Vaswani et al., 2017) can attend to the ∗Equal contribution, sorted alphabetically. 1Our code is publicly available at: https://github.com/facebookresearch/Mask-Predict entire sequence (left and right context) to pre- dict each masked word. We train with a simple masking scheme where the number of masked tar- get tokens is distributed uniformly, presenting the model with both easy (single mask) and difficult (completely masked) examples. Unlike recently proposed insertion models (Gu et al., 2019; Stern et al., 2019), which treat each token as a separate training instance, CMLMs can train from the en- tire sequence in parallel, resulting in much faster training. We also introduce a new decoding algorithm, mask-predict, which uses the order-agnostic na- ture of CMLMs to support highly parallel decod- ing. Mask-predict repeatedly masks out and re- predicts the subset of words in the current trans- lation that the model is least confident about, in contrast to recent parallel decoding translation approaches that repeatedly predict the entire se- quence (Lee et al., 2018). Decoding starts with a completely masked target text, to predict all of the words in parallel, and ends after a constant num- ber of mask-predict cycles. This overall strategy allows the model to repeatedly reconsider …

Content truncated to 500 words