CILex: An Investigation of Context Information for Lexical Substitution Methods

Proceedings of the 29th International Conference on Computational Linguistics, pages 4124–4135 October 12–17, 2022. 4124 CILex: An Investigation of Context Information for Lexical Substitution Methods Sandaru Seneviratne1, Elena Daskalaki1, Artem Lenskiy1, Hanna Suominen1,2 1The Australian National University (ANU) / Canberra, ACT, Australia 2University of Turku / Turku, Finland {sandaru.seneviratne, eleni.daskalaki, artem.lenskiy, hanna.suominen}@anu.edu.au Abstract Lexical substitution, which aims to generate substitutes for a target word given a context, is an important natural language processing task useful in many applications. Due to the paucity of annotated data, existing methods for lexical substitution tend to rely on man- ually curated lexical resources and contextual word embedding models. Methods based on lexical resources are likely to miss relevant substitutes whereas relying only on contextual word embedding models fails to provide ad- equate information on the impact of a substi- tute in the entire context and the overall mean- ing of the input. We proposed CILex, which uses contextual sentence embeddings along with methods that capture additional Context Information complimenting contextual word embeddings for Lexical substitution. This en- sured the semantic consistency of a substi- tute with the target word while maintaining the overall meaning of the sentence. Our ex- perimental comparisons with previously pro- posed methods indicated that our solution is now the state-of-the-art on both the widely used LS07 and CoInCo datasets with P@1 scores of 55.96% and 57.25% for lexical substitution. The implementation of the proposed approach is available at https://github.com/ sandaruSen/CILex under the MIT li- cense. 1 Introduction Lexical substitution is an important Natural Lan- guage Processing (NLP) task, which aims to gener- ate and rank suitable candidate words to replace a given target word, while maintaining the meaning of the given sentence. Lexical substitution is used in a wide range of NLP tasks like data augmenta- tion, paraphrase generation, word sense induction, or text simplification (Shardlow, 2014; Amrami and Goldberg, 2018). Through the years, different approaches have been introduced for lexical substitution but, due to the paucity of annotated data, most of the lexical substitution systems rely on unsupervised methods based on lexical resources or pre-trained language models (Lacerra et al., 2021). Earlier, methods typically relied entirely on manually curated lex- ical resources like WordNet (Miller, 1995). The synonyms obtained from such resources were then ranked based on their suitability evaluated by a similarity metric and predefined rules. Some ap- proaches used vector-based modelling and distri- butional vectors based on syntactic context to ob- tain the most suitable synonyms (Melamud et al., 2015b). Recent advances in contextual language models like Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2019), Embeddings from Language Models (ELMo) (Pe- ters et al., 2018), and XLNet (Yang et al., 2019) have resulted in major breakthroughs in NLP. Be- cause these models carry contextual information and have the ability of context-sensitive modelling of word probabilities, they have achieved the state- of-the-art (SOTA) results in lexical substitution as well. Some recent research efforts have improved lexical substitution by modifying the architecture of contextual embedding models (Zhou …

Content truncated to 500 words