Always Keep your Target in Mind: Studying Semantics and Improving Performance of Neural Lexical Substitution

Proceedings of the 28th International Conference on Computational Linguistics, pages 1242–1255 Barcelona, Spain (Online), December 8-13, 2020 1242 Always Keep your Target in Mind: Studying Semantics and Improving Performance of Neural Lexical Substitution Nikolay Arefyev1, 2, 3, Boris Sheludko1, 2, Alexander Podolskiy1∗, and Alexander Panchenko4 1Samsung Research Center Russia, Moscow, Russia 2Lomonosov Moscow State University, Moscow, Russia 3HSE University, Moscow, Russia 4Skolkovo Institute of Science and Technology, Moscow, Russia narefjev@cs.msu.ru {b.sheludko,a.podolskiy}@samsung.com a.panchenko@skoltech.ru Abstract Lexical substitution, i.e. generation of plausible words that can replace a particular target word in a given context, is an extremely powerful technology that can be used as a backbone of various NLP applications, including word sense induction and disambiguation, lexical relation extraction, data augmentation, etc. In this paper, we present a large-scale comparative study of lexical substitution methods employing both rather old and most recent language and masked language models (LMs and MLMs), such as context2vec, ELMo, BERT, RoBERTa, XLNet. We show that already competitive results achieved by SOTA LMs/MLMs can be further substantially improved if information about the target word is injected properly. Several existing and new target word injection methods are compared for each LM/MLM using both intrinsic evaluation on lexical substitution datasets and extrinsic evaluation on word sense induction (WSI) datasets. On two WSI datasets we obtain new SOTA results. Besides, we analyze the types of semantic relations between target words and their substitutes generated by different models or given by annotators. 1 Introduction Lexical substitution is the task of generating words that can replace a given word in a given textual context. For instance, in the sentence “My daughter purchased a new car” the word car can be substituted by its synonym automobile, but also with co-hyponym bike, or even hypernym motor vehicle while keeping the original sentence grammatical. Lexical substitution has been proven effective in various applications, such as word sense induction (Amrami and Goldberg, 2018), lexical relation extraction (Schick and Sch¨utze, 2020), paraphrase generation, text simplification, textual data augmentation, etc. Note that the preferable type (e.g., synonym, hypernym, co-hyponym, etc.) of generated substitutes depends on the task at hand. The new generation of language and masked language models (LMs/MLMs) based on deep neural networks enabled a profound breakthrough in almost all NLP tasks. These models are commonly used to perform pre-training of deep neural networks, which are then fine-tuned to some final task different from language modeling. However, in this paper we study how the progress in unsupervised pre-training over the last five years affected the quality of lexical substitution. We adapt context2vec (Melamud et al., 2016), ELMo (Peters et al., 2018), BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019) and XLNet (Yang et al., 2019) to solve lexical substitution task without any fine-tuning, but using additional techniques to ensure similarity of substitutes to the target word, which we call target word injection techniques. We provide the first large-scale comparison of various neural LMs/MLMs with several target word injection methods on lexical substitution and WSI tasks. Our research …

Content truncated to 500 words