Towards Better Context-aware Lexical Semantics:Adjusting Contextualized Representations through Static Anchors

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4066–4075, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4066 Towards Better Context-aware Lexical Semantics: Adjusting Contextualized Representations through Static Anchors Qianchu Liu, Diana McCarthy, Anna Korhonen Language Technology Lab, TAL, University of Cambridge, UK ql261@cam.ac.uk, diana@dianamccarthy.co.uk, alk23@cam.ac.uk Abstract One of the most powerful features of con- textualized models is their dynamic embed- dings for words in context, leading to state-of- the-art representations for context-aware lex- ical semantics. In this paper, we present a post-processing technique that enhances these representations by learning a transformation through static anchors. Our method requires only another pre-trained model and no labeled data is needed. We show consistent improve- ment in a range of benchmark tasks that test contextual variations of meaning both across different usages of a word and across different words as they are used in context. We demon- strate that while the original contextual rep- resentations can be improved by another em- bedding space from either contextualized or static models, the static embeddings, which have lower computational requirements, pro- vide the most gains. 1 Introduction Word representations are fundamental in Natural Language Processing (NLP) (Bengio et al., 2003). Recently, there has been a surge of contextualized models that achieve state-of-the-art in many NLP benchmark tasks (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019b; Yang et al., 2019). Even better performance has been reported from fine- tuning or training multiple contextualized models for a specific task such as question answering (De- vlin et al., 2019; Xu et al., 2020). However, little has been explored on directly leveraging the many off-the-shelf pre-trained models to improve task- independent representations for lexical semantics. Furthermore, classic static embeddings are often overlooked in this trend towards contextualized models. As opposed to contextualized embeddings that generate dynamic representations for words in context, static embeddings such as word2vec (Mikolov et al., 2013) assign one fixed representa- tion for each word. Despite being less effective in capturing context-sensitive word meanings, static embeddings still achieve better performance than contextualized embeddings in traditional context- independent lexical semantic tasks including word similarity and analogy (Wang et al., 2019). This suggests that static embeddings have the potential to offer complementary semantic information to en- hance contextualized models for lexical semantics. We bridge the aforementioned gaps and propose a general framework that improves contextualized representations by leveraging other pre-trained con- textualized/static models. We achieve this by using static anchors (the average contextual representa- tions for each word) to transform the original con- textualized model, guided by the embedding space from another model. We assess the overall quality of a model’s lexical semantic representation by two Inter Word tasks that measure relations between dif- ferent words in context. We also evaluate on three Within Word tasks that test the contextual effect from different usages of the same word/word pair. Our method obtains consistent improvement across all these context-aware lexical semantic tasks. We demonstrate the particular strength of leveraging static embeddings, …

Content truncated to 500 words