Combination of Contextualized and Non-Contextualized Layers for Lexical Substitution in French

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6914–6921 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6914 Combination of Contextualized and Non-Contextualized Layers for Lexical Substitution in French K´evin Espasa1, Emmanuel Morin1, Olivier Hamon2 (1) LS2N Nantes University, France (2) Syllabs, Paris, France {firstname.name}@univ-nantes.fr {name}@syllabs.com Abstract Lexical substitution task requires to substitute a target word by candidates in a given context. Candidates must keep meaning and grammatically of the sentence. The task, introduced in the SemEval 2007, has two objectives. The first objective is to find a list of substitutes for a target word. This list of substitutes can be obtained with lexical resources like WordNet or generated with a pre-trained language model. The second objective is to rank these substitutes using the context of the sentence. Most of the methods use vector space models or more recently embeddings to rank substitutes. Embedding methods use high contextualized representation. This representation can be over contextualized and in this way overlook good substitute candidates which are more similar on non-contextualized layers. SemDis 2014 introduced the lexical substitution task in French. We propose an application of the state-of-the-art method based on BERT in French and a novel method using contextualized and non-contextualized layers to increase the suggestion of words having a lower probability in a given context but that are more semantically similar. Experiments show our method increases the BERT based system on the OOT measure but decreases on the BEST measure in the SemDis 2014 benchmark. Keywords: Lexical substitution, French language, Combination contextualized layers 1. Introduction Lexical substitution (McCarthy and Navigli, 2007) is the task of proposing substitution words to replace a target word in a given context. The task can be split into two issues. Firstly, how to find a list of candidates (or synonyms) for the target word and secondly how to validate or invalidate them in a given textual context. For instance, “She paints the bank of the river”, edge can substitute bank while keeping the meaning and the grammaticality of the sentence. Whereas the substitu- tion by the synonym financial institution modifies the meaning of bank. The ranking methods use the context of the sentence to understand the sense of bank to ac- cept edge and reject financial institution. Historically, the list of synonyms for a target word is obtained in lexical resources which contain synonyms like WordNet (Miller, 1992). After picking, candidates are ranked by their quality to replace the target word in its context. With the language models rise using contextual embeddings like BERT, ELMO, candidates are generated by the model with the contextual infor- mation. Then, semantic similarity, between target and candidate of the original sentence and the sentence con- taining the substitution, is computed to compare the im- pact of the substitution on the meaning and choose the best candidate. Lexical substitution is used in many applications like semantic expansion (Han et al., 2020), paraphrasing (Thater et al., 2009), word sense induction (Alagi´c …

Content truncated to 500 words