Embedding Recycling for Language Models

Findings of the Association for Computational Linguistics: EACL 2023, pages 1933–1953 May 2-6, 2023 ©2023 Association for Computational Linguistics Embedding Recycling for Language Models Jon Saad-Falcon1 Amanpreet Singh1 Luca Soldaini1 Mike D’Arcy2 Arman Cohan1,3 Doug Downey1,2 1Allen Institute for Artificial Intelligence (AI2) 2 Northwestern University 3 Yale University {jons, amanpreets, lucas, armanc, dougd}@allenai.org, m.m.darcy@u.northwestern.edu Abstract Real-world applications of neural language models often involve running many different models over the same corpus. The resulting high computational cost has led to interest in techniques that can reuse the contextualized em- beddings produced in previous runs to speed training and inference of future ones. We re- fer to this approach as embedding recycling (ER). While multiple ER techniques have been proposed, their practical effectiveness is still unknown because existing evaluations consider very few models and do not adequately account for overhead costs. We perform an extensive evaluation of ER across eight different models (17 to 900 million parameters) and fourteen tasks in English. We show how a simple ER technique that caches activations from an in- termediate layer of a pretrained model, and learns task-specific adapters on the later layers, is broadly effective. For the best-performing baseline in our experiments (DeBERTa-v2 XL), adding a precomputed cache results in a >90% speedup during training and 87-91% speedup for inference, with negligible impact on accu- racy. Our analysis reveals important areas of future work, and we release code and documen- tation for our experiments at https://github. com/allenai/embeddingrecycling. 1 Introduction Large pretrained language models form the foun- dation of modern NLP, and continue to push the state-of-the-art on a wide range of natural language processing tasks (Devlin et al., 2019; Liu et al., 2019b; Bommasani et al., 2021). Larger models tend to offer superior accuracies (Kaplan et al., 2020), but also entail higher computational costs. The steep computational cost associated with large neural language models slows down experimenta- tion, increases financial barriers to the technology, and contributes to global climate change (Strubell et al., 2019; Dodge et al., 2022). Our work studies how to reduce computational cost for workloads in which many distinct models are run over the same text. For example, a scholarly search tool that helps users find and understand rel- evant literature may run separate models for entity recognition, topic classification, relation extraction, summarization, question answering, and so on over a large corpus of papers. New and improved mod- els for the tasks are developed frequently, necessi- tating additional runs. The need for repeated model runs has also been noted for other applications in previous work, including news applications (Du et al., 2020) and virtual assistants (Wei et al., 2022). Further, repeated runs also occur very frequently during model development, when exploring model variants or executing multiple training epochs. Recent work has introduced ways to reduce com- putational cost in such settings by re-using model activations from one task to speed up other ones (Du et al., 2020; Wei et al., 2022). A pretrained language model’s internal activations form a con- textualized embedding, …

Content truncated to 500 words