Masked Language Model Scoring

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699–2712 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2699 Masked Language Model Scoring Julian Salazar♠ Davis Liang♠ Toan Q. Nguyen♦∗ Katrin Kirchhoff♠ ♠ Amazon AWS AI, USA ♦ University of Notre Dame, USA {julsal,liadavis,katrinki}@amazon.com, tnguye28@nd.edu Abstract Pretrained masked language models (MLMs) require finetuning for most NLP tasks. Instead, we evaluate MLMs out of the box via their pseudo-log-likelihood scores (PLLs), which are computed by masking tokens one by one. We show that PLLs outperform scores from autoregressive language models like GPT-2 in a variety of tasks. By rescoring ASR and NMT hypotheses, RoBERTa reduces an end- to-end LibriSpeech model’s WER by 30% rela- tive and adds up to +1.7 BLEU on state-of-the- art baselines for low-resource translation pairs, with further gains from domain adaptation. We attribute this success to PLL’s unsupervised ex- pression of linguistic acceptability without a left-to-right bias, greatly improving on scores from GPT-2 (+10 points on island effects, NPI licensing in BLiMP). One can finetune MLMs to give scores without masking, enabling com- putation in a single inference pass. In all, PLLs and their associated pseudo-perplexities (PP- PLs) enable plug-and-play use of the growing number of pretrained MLMs; e.g., we use a single cross-lingual model to rescore transla- tions in multiple languages. We release our library for language model scoring at https: //github.com/awslabs/mlm-scoring. 1 Introduction BERT (Devlin et al., 2019) and its improvements to natural language understanding have spurred a rapid succession of contextual language repre- sentations (Yang et al., 2019; Liu et al., 2019; inter alia) which use larger datasets and more involved training schemes. Their success is at- tributed to their use of bidirectional context, often via their masked language model (MLM) objec- tives. Here, a token wt is replaced with [MASK] and predicted using all past and future tokens W\t := (w1, . . . , wt−1, wt+1, . . . , w|W |). ∗Work done during an internship at Amazon AWS AI. Figure 1: To score a sentence, one creates copies with each token masked out. The log probability for each missing token is summed over copies to give the pseudo-log-likelihood score (PLL). One can adapt to the target domain to improve performance, or finetune to score without masks to improve memory usage. In contrast, conventional language models (LMs) predict wt using only past tokens W<t := (w1, . . . , wt−1). However, this allows LMs to es- timate log probabilities for a sentence W via the chain rule (log PLM(W ) = �|W | t=1 log PLM(wt | W<t)), which can be used out of the box to rescore hypotheses in end-to-end speech recognition and machine translation (Chan et al., 2016; Gulcehre et al., 2015), and to evaluate sentences for linguistic acceptability (Lau et al., 2017). Our work studies the corresponding pseudo-log- likelihood scores (PLLs) from MLMs (Wang and Cho, 2019), given by summing the conditional log probabilities log PMLM(wt | W\t) of each sen- …

Content truncated to 500 words