General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3018–3030 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3018 General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference Jingfei Du∗ Myle Ott∗ Haoran Li Xing Zhou Veselin Stoyanov Facebook AI {jingfeidu,myleott,aimeeli,xingz,ves}@fb.com Abstract The state of the art on many NLP tasks is cur- rently achieved by large pre-trained language models, which require a considerable amount of computation. We aim to reduce the infer- ence cost in a setting where many different pre- dictions are made on a single piece of text. In that case, computational cost during inference can be amortized over the different predictions (tasks) using a shared text encoder. We com- pare approaches for training such an encoder and show that encoders pre-trained over multi- ple tasks generalize well to unseen tasks. We also compare ways of extracting fixed- and limited-size representations from this encoder, including pooling features extracted from mul- tiple layers or positions. Our best approach compares favorably to knowledge distillation, achieving higher accuracy and lower computa- tional cost once the system is handling around 7 tasks. Further, we show that through bi- nary quantization, we can reduce the size of the extracted representations by a factor of 16 to store them for later use. The resulting method offers a compelling solution for using large-scale pre-trained models at a fraction of the computational cost when multiple tasks are performed on the same text. 1 Introduction Large pre-trained language models achieve state- of-the-art performance on many Natural Language Processing (NLP) tasks (Peters et al., 2018; Rad- ford et al., 2018; Devlin et al., 2019). However, inference for these models requires significant computational resources, which limits their prac- tical use. Recent trends show that scaling models up (Liu et al., 2019b; Lan et al., 2020; Raffel et al., 2019; Li et al., 2020) in terms of computation still improves end task performance, raising questions ∗Equal contribution. about whether and how the most accurate models can be applied in real-world settings. This computational burden is exacerbated by the need to fine-tune end-to-end a separate model for each task. Since each model has a new set of pa- rameters, none of the computation can be shared by models for different tasks during inference. This is particularly inefficient in real-world settings that require multiple predictions about each input. For example, given a news article, we may want to pre- dict its topic (Zhang et al., 2015), sentiment (Pang and Lee, 2004; Maas et al., 2011; Socher et al., 2013; Zhang et al., 2015), overall text quality (Pitler and Nenkova, 2008), whether it is humorous (Yang et al., 2015) or offensive (Schmidt and Wiegand, 2017; Zampieri et al., 2019) and so on. Knowledge Distillation (KD) is a way of reduc- ing the computation required by large pre-trained LMs (Hinton et al., 2015; Sanh et al., 2019). How- ever, there is a sizeable gap in accuracy between the best models using knowledge distillation and the full fine-tuned models. …

Content truncated to 500 words