Specializing Unsupervised Pretraining Models for Word-Level Semantic Similarity

Proceedings of the 28th International Conference on Computational Linguistics, pages 1371–1383 Barcelona, Spain (Online), December 8-13, 2020 1371 Specializing Unsupervised Pretraining Models for Word-Level Semantic Similarity Anne Lauscher1 Ivan Vuli´c2 Edoardo M. Ponti2 Anna Korhonen2 Goran Glavaˇs1 1Data and Web Science Group, University of Mannheim, Germany 2Language Technology Lab, University of Cambridge, UK 1{anne,goran}@informatik.uni-mannheim.de, 2{iv250,ep490,alk23}@cam.ac.uk Abstract Unsupervised pretraining models have been shown to facilitate a wide range of downstream NLP applications. These models, however, retain some of the limitations of traditional static word embeddings. In particular, they encode only the distributional knowledge available in raw text corpora, incorporated through language modeling objectives. In this work, we complement such distributional knowledge with external lexical knowledge, that is, we integrate the discrete knowl- edge on word-level semantic similarity into pretraining. To this end, we generalize the standard BERT model to a multi-task learning setting where we couple BERT’s masked language modeling and next sentence prediction objectives with an auxiliary task of binary word relation classification. Our experiments suggest that our “Lexically Informed” BERT (LIBERT), specialized for the word-level semantic similarity, yields better performance than the lexically blind “vanilla” BERT on several language understanding tasks. Concretely, LIBERT outperforms BERT in 9 out of 10 tasks of the GLUE benchmark and is on a par with BERT in the remaining one. Moreover, we show consistent gains on 3 benchmarks for lexical simplification, a task where knowledge about word-level semantic similarity is paramount, as well as large gains on lexical reasoning probes. 1 Introduction Unsupervised pretraining models, such as GPT and GPT-2 (Radford et al., 2018; Radford et al., 2019), ELMo (Peters et al., 2018), and BERT (Devlin et al., 2019) yield state-of-the-art performance on a wide range of natural language processing tasks. All these models rely on language modeling (LM) objectives that exploit the knowledge encoded in large text corpora. BERT (Devlin et al., 2019), as one of the current state-of-the-art models, is pretrained on a joint objective consisting of two parts: (1) masked language modeling (MLM), and (2) next sentence prediction (NSP). Through both of these objectives, BERT still consumes only the distributional knowledge encoded by word co-occurrences. While several concurrent research threads are focused on making BERT optimization more robust (Liu et al., 2019) or on imprinting external world knowledge on its representations (Sun et al., 2019; Zhang et al., 2019; Sun et al., 2020; Liu et al., 2020; Peters et al., 2019; Wang et al., 2020, inter alia), no study yet has been dedicated to mitigating a severe limitation that contextualized representations and unsupervised pretraining inherited from static word embeddings: every model that relies on distributional patterns has a tendency to conflate together pure lexical semantic similarity with broader topical relatedness (Schwartz et al., 2015; Mrkˇsi´c et al., 2017). In the past, a plethora of models have been proposed for injecting linguistic constraints (i.e., lexical knowledge) from external resources to static word embeddings (Faruqui et al., 2015; Wieting et al., 2015; Mrkˇsi´c et al., 2017; Ponti et al., 2018, inter alia) …

Content truncated to 500 words