DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 879–895 August 1–6, 2021. ©2021 Association for Computational Linguistics 879 DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations John Giorgi1,5,6 Osvald Nitski2,7 Bo Wang1,4,6,7,† Gary Bader1,3,5,† 1Department of Computer Science, University of Toronto 2Faculty of Applied Science and Engineering, University of Toronto 3Department of Molecular Genetics, University of Toronto 4Department of Laboratory Medicine and Pathobiology, University of Toronto 5Terrence Donnelly Centre for Cellular & Biomolecular Research 6Vector Institute for Artificial Intelligence 7Peter Munk Cardiac Center, University Health Network †Co-senior authors {john.giorgi, osvald.nitski, gary.bader}@mail.utoronto.ca bowang@vectorinstitute.ai Abstract Sentence embeddings are an important com- ponent of many natural language processing (NLP) systems. Like word embeddings, sen- tence embeddings are typically learned on large text corpora and then transferred to var- ious downstream tasks, such as clustering and retrieval. Unlike word embeddings, the highest performing solutions for learning sen- tence embeddings require labelled data, limit- ing their usefulness to languages and domains where labelled data is abundant. In this pa- per, we present DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Represen- tations. Inspired by recent advances in deep metric learning (DML), we carefully design a self-supervised objective for learning univer- sal sentence embeddings that does not require labelled training data. When used to extend the pretraining of transformer-based language models, our approach closes the performance gap between unsupervised and supervised pre- training for universal sentence encoders. Im- portantly, our experiments suggest that the quality of the learned embeddings scale with both the number of trainable parameters and the amount of unlabelled training data. Our code and pretrained models are publicly avail- able and can be easily adapted to new domains or used to embed unseen text.1 1 Introduction Due to the limited amount of labelled training data available for many natural language processing (NLP) tasks, transfer learning has become ubiq- uitous (Ruder et al., 2019). For some time, transfer learning in NLP was limited to pretrained word em- beddings (Mikolov et al., 2013; Pennington et al., 1https://github.com/JohnGiorgi/DeCLUTR 2014). Recent work has demonstrated strong trans- fer task performance using pretrained sentence em- beddings. These fixed-length vectors, often re- ferred to as “universal” sentence embeddings, are typically learned on large corpora and then trans- ferred to various downstream tasks, such as cluster- ing (e.g. topic modelling) and retrieval (e.g. seman- tic search). Indeed, sentence embeddings have be- come an area of focus, and many supervised (Con- neau et al., 2017), semi-supervised (Subramanian et al., 2018; Phang et al., 2018; Cer et al., 2018; Reimers and Gurevych, 2019) and unsupervised (Le and Mikolov, 2014; Jernite et al., 2017; Kiros et al., 2015; Hill et al., 2016; Logeswaran and Lee, 2018) approaches have been proposed. However, the highest performing solutions require labelled data, limiting their usefulness to languages and do- mains where labelled data is abundant. Therefore, closing the performance gap between unsupervised and supervised universal sentence embedding meth- ods is an important goal. Pretraining …

Content truncated to 500 words