An Information Minimization Based Contrastive Learning Model for Unsupervised Sentence Embeddings Learning

Proceedings of the 29th International Conference on Computational Linguistics, pages 4821–4831 October 12–17, 2022. 4821 An Information Minimization Based Contrastive Learning Model for Unsupervised Sentence Embeddings Learning Shaobin Chen1, Jie Zhou2, Yuling Sun1∗and Liang He1 1School of Computer Science and Technology, East China Normal University Shanghai, China 2School of Computer Science, Fudan Univerisity Shanghai, China shaobin_chen@stu.ecnu.edu.cn, jie_zhou@fudan.edu.cn, {ylsun, lhe}@cs.ecnu.edu.cn Abstract Unsupervised sentence embeddings learning has been recently dominated by contrastive learning methods (e.g., SimCSE), which keep positive pairs similar and push negative pairs apart. The contrast operation aims to keep as much information as possible by maximizing the mutual information between positive in- stances, which leads to redundant information in sentence embedding. To address this prob- lem, we present an information minimization based contrastive learning (InforMin-CL) model to retain the useful information and dis- card the redundant information by maximizing the mutual information and minimizing the in- formation entropy between positive instances meanwhile for unsupervised sentence repre- sentation learning. Specifically, we find that information minimization can be achieved by simple contrast and reconstruction objectives. The reconstruction operation reconstitutes the positive instance via the other positive instance to minimize the information entropy between positive instances. We evaluate our model on fourteen downstream tasks, including both su- pervised and unsupervised (semantic textual similarity) tasks. Extensive experimental re- sults show that our InforMin-CL obtains a state-of-the-art performance. Code is made available. 1 1 Introduction How to learn universal sentence embeddings by large-scale pre-trained models (Devlin et al., 2019; Liu et al., 2019), such as BERT, has been studied ex- tensively in the literature (Gao et al., 2021; Reimers et al., 2019). Recently, contrastive learning has been used widely to learn better sentence embed- dings (Meng et al., 2021; Gao et al., 2021). Gener- ally, contrastive learning uses various data augmen- tation methods to generate different views of the input sentences and samples positive instances and ∗ Yuling Sun is the corresponding authors of this paper. 1https://github.com/Bin199/InforMin-CL Table 1: Training texts may contain various kinds of redundant information, such as stop words, restatement, capitalization, and hyphen. Original Where is the party, it sounds great. Stop words Where is the party, it sounds great. Restatement The party sounds great, where is it. Capitalization Where Is The Party, It Sounds Great. Hyphen Where-is-the-party, it-sounds-great. negative instances from views. Contrastive learn- ing aims to learn effective embeddings by pulling positive instances together and pushing positive and negative instances apart. This operation focus on maximizing the mutual information between positive instances to retain as much information as possible, which includes both the useful and use- less information. Previous studies like (Meng et al., 2021; Gao et al., 2021) ignore the redundant infor- mation stored in views, which has a bad impact on the performance of downstream tasks, as proved by (Achille and Soatto, 2018; Tian et al., 2020). Table 1 gives an example of redundant infor- mation stored in training texts. In training texts, the sentences contain much redundant information which is not favorable to the downstream task. …

Content truncated to 500 words