BERT-based Lexical Substitution

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3368–3373 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3368 BERT-based Lexical Substitution Wangchunshu Zhou1 ∗ Tao Ge2 Ke Xu1 Furu Wei2 Ming Zhou2 1Beihang University, Beijing, China 2Microsoft Research Asia, Beijing, China zhouwangchunshu@buaa.edu.cn, kexu@nlsde.buaa.edu.cn {tage, fuwei, mingzhou}@microsoft.com Abstract Previous studies on lexical substitution tend to obtain substitute candidates by finding the tar- get word’s synonyms from lexical resources (e.g., WordNet) and then rank the candidates based on its contexts. These approaches have two limitations: (1) They are likely to over- look good substitute candidates that are not the synonyms of the target words in the lexical re- sources; (2) They fail to take into account the substitution’s influence on the global context of the sentence. To address these issues, we propose an end-to- end BERT-based lexical substitution approach which can propose and validate substitute can- didates without using any annotated data or manually curated resources. Our approach first applies dropout to the target word’s em- bedding for partially masking the word, al- lowing BERT to take balanced consideration of the target word’s semantics and contexts for proposing substitute candidates, and then validates the candidates based on their sub- stitution’s influence on the global contextual- ized representation of the sentence. Exper- iments show our approach performs well in both proposing and ranking substitute candi- dates, achieving the state-of-the-art results in both LS07 and LS14 benchmarks. 1 Introduction Lexical substitution (McCarthy and Navigli, 2007) aims to replace a target word in a sentence with a substitute word without changing the meaning of the sentence, which is useful for many Natural Language Processing (NLP) tasks like text simpli- fication and paraphrase generation. One main challenge in this task is proposing substitutes that not only are semantically consis- tent with the original target word and fits in the ∗This work was done during the first author’s internship at Microsoft Research Asia. Sentence The wine he sent to me as my birthday gift is too strong to drink. WordNet hard,+solid,+s7ff,+firm BERT+(keep+target+word) stronger,+strongly,+hard,+much BERT+(mask+target+word) hot,+thick,+sweet,+much BERT-(embedding-dropout) tough,+powerful,+potent,+hard Sentence The wine he sent to me as my birthday gift is too strong to drink. The wine he sent to me as my birthday gift is too hot (0.81) to drink. (0.86) The wine he sent to me as my birthday gift is too tough (0.91) to drink. (0.92) The wine he sent to me as my birthday gift is too powerful (0.91) to drink. (0.93) (a) (b) Figure 1: (a) WordNet and original BERT cannot pro- pose the valid substitute powerful in their top-K results but applying target word embedding dropout enables BERT to propose it; (b) Undesirable substitutes (e.g., hot, tough) tend to change the contextualized represen- tation of the sentence more than good substitutes (e.g., powerful). The numbers after the words are the co- sine similarity of the words’ contextualized vector to the original target words; while the numbers after the sentence are the similarity of …

Content truncated to 500 words