Simple and Effective Multi-Token Completion from Masked Language Models

Findings of the Association for Computational Linguistics: EACL 2023, pages 2356–2369 May 2-6, 2023 ©2023 Association for Computational Linguistics Simple and Effective Multi-Token Completion from Masked Language Models Oren Kalinsky * Amazon orenk@amazon.com Guy Kushilevitz * Amazon guyk@amazon.com Alex Libov Amazon alibov@amazon.com Yoav Goldberg Bar Ilan University yoav.goldberg@gmail.com Abstract Pre-trained neural masked language models are often used for predicting a replacement token for a given sequence position, in a cloze-like task. However, this usage is re- stricted to predicting a single token, from a relatively small pre-trained vocabulary. Recent Sequence2Sequence pre-trained LMs like T5 do allow predicting multi-token completions, but are more expensive to train and run. We show that pre-trained masked language models can be adapted to produce multi-token com- pletions, with only a modest addition to their parameter count. We propose two simple adap- tation approaches, trading parameter counts for accuracy. The first method generates multi- token completions from a conditioned RNN. It has a very low parameter count and achieves competitive results. The second method is even simpler: it adds items corresponding to multi-token units to the output prediction ma- trix. While being higher in parameter count than the RNN method, it also surpasses current state-of-the-art multi-token completion mod- els, including T5-3B, while being significantly more parameter efficient. We demonstrate that our approach is flexible to different vocabular- ies and domains and can effectively leverage ex- isting pre-trained models available in different domains. Finally, a human evaluation further validates our results and shows that our solution regularly provides valid completions, as well as reasonable correctness for factual-sentence completions. 1 Introduction Multi-Token-Completion (MTC) is the task of fill- ing masked sentences with a sequence of tokens such that the completed sentence is probable and coherent. e.g., for the masked sentence "The 46th president of the US, [MASK], was elected in 2020" *Authors contributed equally. a good completion would be any of "Biden", "Joe Biden", "president Biden" and more. While Masked Language Models (MLMs) such as BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) successfully deal with a simpler vari- ation of this task (single-token completion) these models were pre-trained on a limited vocabulary, not containing multi-word phrases. It is technically possible to complete numerous tokens simultane- ously with MLMs by introducing a sequence of [MASK] tokens, but there are no clear methods of conditioning the tokens on each other, and, more importantly, the length of the completion needs to be pre-determined. Expanding MLMs’ effec- tive completion vocabulary – the actual vocabulary they are capable of completing well – will help a recent line of work using MLMs to extract knowl- edge/information from corpora (Jiang et al., 2020; Petroni et al., 2019; Kushilevitz et al., 2020). Most existing works using MLMs to complete sentences either avoid the problem by limiting the completions to single tokens or use sub-optimal heuristics for MTC (e.g. presetting the length of the sequence and filling one token at a time). A triv- ial solution is to increase the number of tokens …

Content truncated to 500 words