Obtaining Better Static Word Embeddings Using Contextual Embedding Models

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5241–5253 August 1–6, 2021. ©2021 Association for Computational Linguistics 5241 Obtaining Better Static Word Embeddings Using Contextual Embedding Models Prakhar Gupta EPFL, Switzerland prakhar.gupta@epfl.ch Martin Jaggi EPFL, Switzerland martin.jaggi@epfl.ch Abstract The advent of contextual word embeddings— representations of words which incorporate se- mantic and syntactic information from their context—has led to tremendous improvements on a wide variety of NLP tasks. However, recent contextual models have prohibitively high computational cost in many use-cases and are often hard to interpret. In this work, we demonstrate that our proposed distilla- tion method, which is a simple extension of CBOW-based training, allows to significantly improve computational efficiency of NLP ap- plications, while outperforming the quality of existing static embeddings trained from scratch as well as those distilled from previ- ously proposed methods. As a side-effect, our approach also allows a fair comparison of both contextual and static embeddings via standard lexical evaluation tasks. 1 Introduction Word embeddings—representations of words which reflect semantic and syntactic information carried by them are ubiquitous in Natural Language Processing. Static word representation models such as GLOVE (Pennington et al., 2014), CBOW, SKIPGRAM (Mikolov et al., 2013) and SENT2VEC (Pagliardini et al., 2018) obtain stand-alone rep- resentations which do not depend on their sur- rounding words or sentences (context). Contex- tual embedding models (Devlin et al., 2019; Peters et al., 2018; Liu et al., 2019; Radford et al., 2019; Schwenk and Douze, 2017) on the other hand, em- bed the contextual information as well into the word representations making them more expressive than static word representations in most use-cases. While recent progress on contextual embeddings has been tremendously impactful, static embed- dings still remain fundamentally important in many scenarios as well: • Even when ignoring the training phase, the computational cost of using static word em- beddings is typically tens of millions times lower than using standard contextual embed- ding models1, which is particularly important for latency-critical applications and on low- resource devices, and in view of environmen- tal costs of NLP models (Strubell et al., 2019). • Many NLP tasks inherently rely on static word embeddings (Shoemark et al., 2019), for ex- ample for interpretability, or e.g. in research in bias detection and removal (Kaneko and Bollegala, 2019; Gonen and Goldberg, 2019; Manzini et al., 2019) and analyzing word vec- tor spaces (Vulic et al., 2020) or other metrics which are non-contextual by choice. • Static word embeddings can complement con- textual word embeddings, for separating static from contextual semantics (Barsalou, 1982; Rubio-Fern´andez, 2008), or for improving joint embedding performance on downstream tasks (Alghanmi et al., 2020). We also refer the reader to this article2 illustrating several down-sides of using BERT-like models over static embedding models for non-specialist users. Indeed, we can see continued prevalence of static word embeddings in industry and research areas including but not limited to medicine (Zhang et al., 2019; Karadeniz and …

Content truncated to 500 words