An Unsupervised Sentence Embedding Method by Mutual Information Maximization

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1601–1610, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1601 An Unsupervised Sentence Embedding Method by Mutual Information Maximization Yan Zhang∗1,2 Ruidan He∗2 Zuozhu Liu†3 Kwan Hui Lim1 Lidong Bing2 1Singapore University of Technology and Design 2DAMO Academy, Alibaba Group 3ZJU-UIUC Institue yan zhang@mymail.sutd.edu.sg, ruidan.he@alibaba-inc.com zuozhuliu@intl.zju.edu.cn, kwanhui lim@sutd.edu.sg l.bing@alibaba-inc.com Abstract BERT is inefficient for sentence-pair tasks such as clustering or semantic search as it needs to evaluate combinatorially many sen- tence pairs which is very time-consuming. Sentence BERT (SBERT) attempted to solve this challenge by learning semantically mean- ingful representations of single sentences, such that similarity comparison can be easily accessed. However, SBERT is trained on cor- pus with high-quality labeled sentence pairs, which limits its application to tasks where la- beled data is extremely scarce. In this paper, we propose a lightweight extension on top of BERT and a novel self-supervised learning ob- jective based on mutual information maximiza- tion strategies to derive meaningful sentence embeddings in an unsupervised manner. Un- like SBERT, our method is not restricted by the availability of labeled data, such that it can be applied on different domain-specific cor- pus. Experimental results show that the pro- posed method significantly outperforms other unsupervised sentence embedding baselines on common semantic textual similarity (STS) tasks and downstream supervised tasks. also outperforms SBERT in a setting whereIt in-domain labeled data is not available, and achieves performance competitive with super- vised methods on various tasks. 1 Introduction BERT-based pretrained language models (Devlin et al.; Liu et al., 2019) have set new state-of- the-art performance on various downstream NLP tasks. However, they are inefficient for sentence- pair regression tasks such as clustering or semantic search because they need to evaluate combinatori- ally many sentence pairs during inference, which will result in a massive computational overhead. ∗ Equally Contributed. This work was done when Yan Zhang was an intern at DAMO Academy, Alibaba Group. †Corresponding author. For example, finding the most similar pair in a col- (lection of 10k sentences requires about 50 million �10k 2 � ) inference computations with BERT, which requires about 65 hours on a V100 GPU (Reimers and Gurevych, 2019). Much previous work attempted to address this problem by learning semantically meaningful rep- resentations for each sentence, such that similar- ity measures like cosine distance can be easily evaluated for sentence-pair regression tasks. The straightforward way to derive a fixed-size sentence embedding from BERT-based models is to aver- age the token representations at the last layer or using the output of the [CLS] token. Reimers and Gurevych (2019) showed that both approaches yield rather unsatisfactory sentence embeddings. They proposed a model, Sentence-BERT (SBERT), to further fine-tune BERT on natural language in- ference (NLI) tasks with labeled sentence pairs and achieved state-of-the-art performance on many semantic textual similarity tasks. However, such improvements are induced by high-quality supervi- sion, and we find that their performance is degraded where labeled data of the target task is …

Content truncated to 500 words