ParaLS: Lexical Substitution via Pretrained Paraphraser

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3731–3746 July 9-14, 2023 ©2023 Association for Computational Linguistics ParaLS: Lexical Substitution via Pretrained Paraphraser Jipeng Qiang and Kang Liu and Yun Li ∗and Yunhao Yuan and Yi Zhu ∗ School of Information Engineering, Yangzhou University, Yangzhou, China {jpqiang, liyun, yhyuan, zhuyi}@yzu.edu.cn, liukang@stu.yzu.edu.cn Abstract Lexical substitution (LS) aims at finding appro- priate substitutes for a target word in a sen- tence. Recently, LS methods based on pre- trained language models have made remarkable progress, generating potential substitutes for a target word through analysis of its contextual surroundings. However, these methods tend to overlook the preservation of the sentence’s meaning when generating the substitutes. This study explores how to generate the substitute candidates from a paraphraser, as the gener- ated paraphrases from a paraphraser contain variations in word choice and preserve the sen- tence’s meaning. Since we cannot directly gen- erate the substitutes via commonly used decod- ing strategies, we propose two simple decoding strategies that focus on the variations of the tar- get word during decoding. Experimental results show that our methods outperform state-of-the- art LS methods based on pre-trained language models on three benchmarks. 1 Introduction Lexical substitution (LS) in context (Hintz and Biemann, 2016; Zhou et al., 2019; Arefyev et al., 2020) is an extremely powerful technology that can be used as a backbone of various NLP applica- tions such as writing assistance (Lee et al., 2021), word sense disambiguation (McCarthy, 2002), and lexical simplification (Paetzold and Specia, 2016; Qiang et al., 2021a,b). Compared with traditional LS methods based on linguistic databases (e.g., WordNet) (Hassan et al., 2007; Yuret, 2007) or word embedding models (Melamud et al., 2015a,b), LS methods based on pretrained language models have made remarkable progress in generating sub- stitutes by considering the context (Zhou et al., 2019; Qiang et al., 2021a; Michalopoulos et al., 2022; Seneviratne et al., 2022). These methods feed the sentence into BERT (Devlin et al., 2018) ∗Corresponding author. or XLNet (Yang et al., 2019) to obtain the top prob- ability words corresponding to the target word as the substitute candidates. However, they have the following two limitations. (1) The predictability of words is greatly influ- enced by the surrounding context, with little re- gard for preserving the sentence’s meaning. As illustrated in Table 1, the utilization of pretrained models often leads to the generation of ill-suited words, such as "wet", "flat" and "cold", due to their contextual relevance and similarity to the target word. (2) The utilization of subword techniques in pre- trained models precludes the selection of multi- token words as substitutes, as they only generate the most probable single tokens. For instance, the words "desiccated" and "facilitated" would not be offered as a substitution for the target word "dry" as seen in Table 1. To address the limitations mentioned above, we study how to generate substitutes via paraphrase modeling. Recent neural paraphrasers based on encoder-decoder framework (Wieting and Gimpel, 2017; Hu …

Content truncated to 500 words