GeneSis: A Generative Approach to Substitutes in Context

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10810–10823 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10810 GENESIS: A Generative Approach to Substitutes in Context Caterina Lacerra Sapienza University of Rome lacerra@di.uniroma1.it Rocco Tripodi University of Bologna rocco.tripodi@unibo.it Roberto Navigli Sapienza University of Rome navigli@diag.uniroma1.it Abstract The lexical substitution task aims at generat- ing a list of suitable replacements for a tar- get word in context, ideally keeping the mean- ing of the modified text unchanged. While its usage has increased in recent years, the paucity of annotated data prevents the finetun- ing of neural models on the task, hindering the full fruition of recently introduced power- ful architectures such as language models. Fur- thermore, lexical substitution is usually evalu- ated in a framework that is strictly bound to a limited vocabulary, making it impossible to credit appropriate, but out-of-vocabulary, sub- stitutes. To assess these issues, we propose GENESIS (Generating Substitutes in contexts), the first generative approach to lexical substi- tution. Thanks to a seq2seq model, we gen- erate substitutes for a word according to the context it appears in, attaining state-of-the- art results on different benchmarks. More- over, our approach allows silver data to be produced for further improving the perfor- mances of lexical substitution systems. Along with an extensive analysis of GENESIS results, we also present a human evaluation of the generated substitutes in order to assess their quality. We release the fine-tuned models, the generated datasets and the code to repro- duce the experiments at https://github. com/SapienzaNLP/genesis. 1 Introduction The lexical substitution task (McCarthy and Nav- igli, 2009) requires a system to provide adequate replacements for a target word in a given context. Through the years, two lexical substitution vari- ants have been proposed, i.e., candidates ranking and substitutes prediction (Melamud et al., 2015). While the former aims at ranking a list of prede- fined candidate substitutes for a word in a given context, the latter is more challenging, requiring a system to output a sorted list of replacements without any predefined substitutes inventory. Al- though it is not explicitly required by either of the two tasks, a good substitution system is expected to capture the semantics of its input and implic- itly perform a soft disambiguation. For example, denoting bright as target word in the context sen- tence "She is a bright student", we expect a good substitution system to provide a set of substitutes closer to {intelligent, clever, smart} than to {lumi- nous, clear, light}. Thanks to this implicit disam- biguation capability, lexical substitution has shown its usefulness in several scenarios, such as word sense induction (Ba¸skaya et al., 2013; Amrami and Goldberg, 2018; Arefyev et al., 2019), data aug- mentation (Jia et al., 2019; Arefyev et al., 2020), word sense disambiguation (Hou et al., 2020) and semantic role labeling (Bingel et al., 2018). However, despite having been employed in nu- merous downstream tasks, the lexical substitution task still presents unresolved issues that need to be addressed. First, the …

Content truncated to 500 words