Unsupervised Lexical Substitution with Decontextualised Embeddings

Proceedings of the 29th International Conference on Computational Linguistics, pages 4172–4185 October 12–17, 2022. 4172 Unsupervised Lexical Substitution with Decontextualised Embeddings Takashi Wada1,2 Timothy Baldwin1,3 Yuji Matsumoto2 Jey Han Lau1 1 School of Computing and Information Systems, The University of Melbourne 2 RIKEN Center for Advanced Intelligence Project (AIP) 3 Department of Natural Language Processing, MBZUAI twada@student.unimelb.edu.au, tb@ldwin.net yuji.matsumoto@riken.jp, jeyhan.lau@gmail.com Abstract We propose a new unsupervised method for lexical substitution using pre-trained language models. Compared to previous approaches that use the generative capability of language mod- els to predict substitutes, our method retrieves substitutes based on the similarity of contextu- alised and decontextualised word embeddings, i.e. the average contextual representation of a word in multiple contexts. We conduct exper- iments in English and Italian, and show that our method substantially outperforms strong baselines and establishes a new state-of-the-art without any explicit supervision or fine-tuning. We further show that our method performs par- ticularly well at predicting low-frequency sub- stitutes, and also generates a diverse list of sub- stitute candidates, reducing morphophonetic or morphosyntactic biases induced by article– noun agreement.1 1 Introduction There has been growing interest in developing auto- matic writing support systems to assist humans to write documents. One relevant task to this research goal is lexical substitution, where given a target word and its surrounding context, a system sug- gests a list of word substitutions that can replace the target word without changing its core meaning. For instance, given the target word great and the context He is a great artist, the model might sug- gest alternative words such as outstanding, terrific, or distinguished. Writers can use such suggestions to improve the fluency of their writing, reduce lexi- cal repetition, or search for better expressions that represent their ideas more creatively. As with other NLP tasks, recent studies have shown that masked language models such as BERT (Devlin et al., 2019) perform very well on lexical substitution, even without any task-specific super- vision. A common approach is to employ language 1Code is available at: https://github.com/ twadada/lexsub_decontextualised. models as generative models and predict substitutes based on their generative capability. However, this approach has some limitations. First, it is extremely difficult for language models to predict rare words — especially those that are segmented into multi- ple subword tokens — since the models inevitably assign them very low probabilities. Second, word prediction is highly affected by morphosyntactic constraints from the surrounding context, which overshadows the (arguably more important) ques- tion of semantic fit. For instance, if the target word is increase in the context ... with an increase in ..., language models tend to suggest words that also start with a vowel sound due to the presence of an before the target word, missing other possible substitutes such as hike or boost. In fact, this prob- lem is even more pronounced in languages where words have grammatical gender (e.g. Italian nouns) or a high degree of inflection (e.g. Japanese verbs). In this paper, we propose a new approach that explicitly …

Content truncated to 500 words