PAUSE: Positive and Annealed Unlabeled Sentence Embedding

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10096–10107 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10096 PAUSE: Positive and Annealed Unlabeled Sentence Embedding Lele Cao1 Emil Larsson1,2 Vilhelm von Ehrenheim1 Dhiana Deva Cavalcanti Rocha1 Anna Martin1 Sonja Horn1 1Motherbrain, EQT Group, Stockholm, Sweden 2Modulai, Stockholm, Sweden {lele.cao,vilhelm.vonehrenheim}@eqtpartners.com {dhiana.deva,anna.martin}@eqtpartners.com emil@modulai.io sonja@eqtventures.com Abstract Sentence embedding refers to a set of effec- tive and versatile techniques for converting raw text into numerical vector representations that can be used in a wide range of natural lan- guage processing (NLP) applications. The ma- jority of these techniques are either supervised or unsupervised. Compared to the unsuper- vised methods, the supervised ones make less assumptions about optimization objectives and usually achieve better results. However, the training requires a large amount of labeled sen- tence pairs, which is not available in many in- dustrial scenarios. To that end, we propose a generic and end-to-end approach – PAUSE (Positive and Annealed Unlabeled Sentence Embedding), capable of learning high-quality sentence embeddings from a partially labeled dataset. We experimentally show that PAUSE achieves, and sometimes surpasses, state-of- the-art results using only a small fraction of labeled sentence pairs on various benchmark tasks. When applied to a real industrial use case where labeled samples are scarce, PAUSE encourages us to extend our dataset without the burden of extensive manual annotation work. 1 Introduction A sentence embedding is a numerical represen- tation used to describe the meaning of an entire sentence. Embeddings of this type are becoming increasingly important for many downstream tasks in the language understanding domain, such as sim- ilarity or sentiment analysis. Some earlier methods, like GloVe (Pennington et al., 2014), BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) pool directly from underlying token-level embeddings to create a sentence representation. Recently, these pooling strategies have been challenged by vari- ous parameterized policies that can be optimized on domain specific tasks. The majority of these are either unsupervised or supervised. While unsu- pervised methods only utilize unlabeled sentences, supervised methods can quickly customize the em- beddings by using domain specific labels. As a consequence, supervised methods make less as- sumptions about optimization objectives and usu- ally achieve better results. However, supervised training requires a large amount of labeled sen- tence pairs, which is usually unavailable. In many real scenarios, the dataset turns out to be positive- unlabeled (i.e. PU dataset), where the majority is unlabeled and the rest of the samples are labeled as positive. The methods that enable learning binary classifiers on PU datasets are called PU learning. To bridge the gap between supervised and unsuper- vised approaches, we incorporate state-of-the-art PU learning with the general supervised sentence embedding approaches, proposing a novel method – PAUSE (Positive and Annealed Unlabeled Sen- tence Embedding)1. The main highlights of PAUSE include: (1) good sentence embeddings can be learned from datasets with only a few positive labels; (2) it can be trained in an end-to-end fashion; (3) it can be directly applied …

Content truncated to 500 words