Contextual Embeddings: When Are They Worth It?

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2650–2663 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2650 Contextual Embeddings: When Are They Worth It? Simran Arora∗, Avner May∗, Jian Zhang, Christopher R´e Stanford Univeristy {simarora, avnermay, zjian, chrismre}@cs.stanford.edu Abstract We study the settings for which deep con- textual embeddings (e.g., BERT) give large improvements in performance relative to classic pretrained embeddings (e.g., GloVe), and an even simpler baseline—random word embeddings—focusing on the impact of the training set size and the linguistic properties of the task. Surprisingly, we find that both of these simpler baselines can match contextual embeddings on industry-scale data, and often perform within 5 to 10% accuracy (absolute) on benchmark tasks. Furthermore, we iden- tify properties of data for which contextual embeddings give particularly large gains: lan- guage containing complex structure, ambigu- ous word usage, and words unseen in training. 1 Introduction In recent years, rich contextual embeddings such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2018) have enabled rapid progress on benchmarks like GLUE (Wang et al., 2019a) and have seen widespread industrial use (Pandu Nayak, 2019). However, these methods require significant com- putational resources (memory, time) during pre- training, and during downstream task training and inference. Thus, an important research problem is to understand when these contextual embeddings add significant value vs. when it is possible to use more efficient representations without significant degradation in performance. As a first step, we empirically compare the per- formance of contextual embeddings with classic embeddings like word2vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014). To further understand what performance gains are attributable to improved embeddings vs. the powerful down- stream models that leverage them, we also com- pare with a simple baseline—fully random embed- ∗Equal contribution. dings—which encode no semantic or contextual information whatsoever. Surprisingly, we find that in highly optimized production tasks at a major technology company, both classic and random em- beddings have competitive (or even slightly better!) performance than the contextual embeddings.1,2 To better understand these results, we study the properties of NLP tasks for which contextual em- beddings give large gains relative to non-contextual embeddings. In particular, we study how the amount of training data, and the linguistic prop- erties of the data, impact the relative performance of the embedding methods, with the intuition that contextual embeddings should give limited gains on data-rich, linguistically simple tasks. In our study on the impact of training set size, we find in experiments across a range of tasks that the performance of the non-contextual embeddings (GloVe, random) improves rapidly as we increase the amount of training data, often attaining within 5 to 10% accuracy of BERT embeddings when the full training set is used. This suggests that for many tasks these embeddings could likely match BERT given sufficient data, which is precisely what we ob- serve in our experiments with industry-scale data. Given the computational overhead of contextual embeddings, this exposes important trade-offs …

Content truncated to 500 words