Inference Strategies for Machine Translation with Conditional Masking

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5774–5782, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5774 Inference Strategies for Machine Translation with Conditional Masking Julia Kreutzer and George Foster and Colin Cherry Google Research {jkreutzer,fosterg,colincherry}@google.com Abstract Conditional masked language model (CMLM) training has proven successful for non- autoregressive and semi-autoregressive se- quence generation tasks, such as machine translation. Given a trained CMLM, however, it is not clear what the best inference strategy is. We formulate masked inference as a factor- ization of conditional probabilities of partial sequences, show that this does not harm per- formance, and investigate a number of simple heuristics motivated by this perspective. We identify a thresholding strategy that has advan- tages over the standard “mask-predict” algo- rithm, and provide analyses of its behavior on machine translation tasks. 1 Introduction The widely successful masked language model- ing paradigm popularized by BERT (Devlin et al., 2019) has recently been adapted to conditional masked language model (CMLM) training for semi- autoregressive sequence generation (Ghazvinine- jad et al., 2019), where model predictions are con- ditioned on the complete input sequence and the observed (non-masked) portion of the output se- quence. The CMLM’s simplicity and its clear links to the very active field of linguistic repre- sentation learning are advantages over its semi- autoregressive competitors, such as iterative re- finement of token sequences (Lee et al., 2018), refinement of non-linguistic intermediate represen- tations (Kaiser et al., 2018; Shu et al., 2020) and learning to predict parallel edit operations (Stern et al., 2019; Gu et al., 2019). It is not obvious how to best perform inference with the CMLM. Starting from a partially-observed output sequence, the optimal choice to complete it within a single step would be to generate the most likely token at each unobserved (masked) po- sition independently. However, it is less clear how to progress from an initial, completely masked se- quence to a final hypothesis semi-autoregressively over a number of steps, with each successive step unmasking new context for the next. This requires not only ordering the tokens for generation, but also making decisions about how many tokens to simultaneously predict in each step. Ghazvininejad et al. (2019) propose the mask- predict algorithm, which iteratively generates fresh model predictions for all masked positions, and then unmasks a predefined number of the most likely predictions. Given a fixed number of itera- tions, a decaying schedule determines how many predictions to unmask in each iteration. Each suc- cessive iteration provides mode-breaking (Gu et al., 2018) context for the next. By fixing the number of iterations, this approach allows for constant-time semi-autoregressive decoding. The fixed-iteration strategy is very practical and has yielded empirical success in a range of machine translation experiments, but there is no guarantee that it is optimal. The tokens to be unmasked on a given iteration are all predicted independently, and therefore might contain repeated words, or words with low model confidence. These issues can be mitigated by later re-masking a …

Content truncated to 500 words