Weight Poisoning Attacks on Pretrained Models

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 263–274 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 263 Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order Yi Liao, Xin Jiang, Qun Liu Huawei Noah’s Ark Lab {liaoyi9, jiang.xin, qun.liu}@huawei.com Abstract Masked language model and autoregressive language model are two types of language models. While pretrained masked language models such as BERT (Devlin et al., 2019) overwhelm the line of natural language un- derstanding (NLU) tasks, autoregressive lan- guage models such as GPT (Radford et al., 2018) are especially capable in natural lan- guage generation (NLG). In this paper, we pro- pose a probabilistic masking scheme for the masked language model, which we call prob- abilistically masked language model (PMLM). We implement a specific PMLM with a uni- form prior distribution on the masking ratio named u-PMLM. We prove that u-PMLM is equivalent to an autoregressive permutated lan- guage model. One main advantage of the model is that it supports text generation in ar- bitrary order with surprisingly good quality, which could potentially enable new applica- tions over traditional unidirectional generation. Besides, the pretrained u-PMLM also outper- forms BERT on a set of downstream NLU tasks. 1 Introduction Large-scale pretrained language models (Raffel et al., 2019; Wang et al., 2019; Lan et al., 2019; Liu et al., 2019; Jiao et al., 2019) have drawn lots of research attention as these models have brought significant improvements to many NLU and NLG tasks. As a major category of pretrained language models, masked language model (MLM) (Devlin et al., 2019; Joshi et al., 2019) is trained using a de- noising autoencoding objective. In a typical MLM, some tokens in a sentence are replaced by a special token [MASK]. The training objective is to predict the original tokens that are masked in the sentence. As the first large-scale pretrained masked language model, BERT chooses to mask 15% of the tokens in sentences randomly. Following BERT, various The wolf has an extraordinary speed , and it can often jump from a spot quick enough to escape a spot already occupied by an adult wolf . Unlike the brown and black bear , where it is easily distracted by wolves , the gray fox does not run over a wolf , and is often driven mad . Hav- ing jumps with high speed that breaks the wolf ’ s legs before it is run over , a grey wolf could defend itself against an adult of other species as the best predator at any time . The black bear may kill packs of four lazy , though the gray fox can inflict significant wounds on a dog . Figure 1: A piece of text generated by a PMLM in ran- dom order. The bolded words, which compose the in- put sentence “The quick brown fox jumps over the lazy dog”, are distributed across the paragraph with a prede- fined length. The blank spaces are filled by …

Content truncated to 500 words