Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models

Findings of the Association for Computational Linguistics: ACL 2022, pages 1864 - 1874 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Sentence-T5 (ST5): Scalable Sentence Encoders from Pre-trained Text-to-Text Models Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith B. Hall, Daniel Cer, Yinfei Yang Google Research Mountain View, CA Abstract We provide the first exploration of sen- tence embeddings from text-to-text transform- ers (T5) including the effects of scaling up sentence encoders to 11B parameters. Sen- tence embeddings are broadly useful for lan- guage processing tasks. While T5 achieves impressive performance on language tasks, it is unclear how to produce sentence embed- dings from encoder-decoder models. We in- vestigate three methods to construct Sentence- T5 (ST5) models: two utilize only the T5 encoder and one using the full T5 encoder- decoder. We establish a new sentence represen- tation transfer benchmark, SentGLUE, which extends the SentEval toolkit to nine tasks from the GLUE benchmark (Wang et al., 2018). Our encoder-only models outperform the previous best models on both SentEval and SentGLUE transfer tasks, including semantic textual simi- larity (STS). Scaling up ST5 from millions to billions of parameters shown to consistently improve performance. Finally, our encoder- decoder method achieves a new state-of-the- art on STS when using sentence embeddings.1 1 Introduction Sentence embeddings providing compact meaning representations that are broadly useful for a vari- ety of language processing tasks include classifi- cation, question-answering, semantic retrieval, bi- text mining, and semantic similarity tasks. We explore sentence embeddings from a new fam- ily of pre-trained models: Text-to-Text Transfer Transformer (T5) (Raffel et al., 2020). Unlike encoder-only models, which use a transformer en- coder to predict random masked tokens, T5 uses an encoder-decoder architecture and a generative span corruption pre-training task. T5 models can be scaled up to hundreds of billions of parameters 1Our models are released at https://tfhub.dev/ google/collections/sentence-t5/1. Figure 1: Scaling up our ST5 model size improves per- formance on SentEval (left) and STS (right). Transfer STS ST5-EncDec (11B params) 90.46 84.94 ST5-Enc (11B params) 91.63 84.96 SimCSE-RoBERTa (large) (Gao et al., 2021)2 90.23 83.76 SBERT (large) (Reimers and Gurevych, 2019) 87.69 76.55 USE (Cer et al., 2018) 85.10 71.22 InferSent (Conneau et al., 2017) 85.59 65.01 Table 1: ST5 versus notable sentence embedding mod- els on SentEval tasks. The reported numbers are the average of transfer tasks (classification accuracy) and STS tasks (spearman correlation). (Fedus et al., 2021) and have achieved state-of-the- art performance on a broad range of NLP tasks including Generalized Language Understanding Evaluation (GLUE) (Wang et al., 2018) and Super- GLUE (Wang et al., 2019). However, it is difficult to efficiently apply T5 to some tasks such as re- trieval or clustering. To score retrieval candidates, T5 would need to perform full inference with cross- attention on each query-candidate pair. In contrast, sentence embeddings allow for efficient retrieval and clustering (Gillick et al., 2018; Reimers and Gurevych, 2019; Yang et al., 2020). As shown in Figure 2, we explore three ways of turning a pre-trained T5 …

Content truncated to 500 words