ConIsI: A Contrastive Framework with Inter-sentence Interaction for Self-supervised Sentence Representation

ConIsI: A Contrastive Framework with Inter-sentence Interaction for Self-supervised Sentence Representation Meng Sun Dalian University of Technology sunmeng20@mail.dlut.edu.cn Degen Huang∗ Dalian University of Technology huangdg@dlut.edu.cn Abstract Learning sentence representation is a fundamental task in natural language processing and has been studied extensively. Recently, many works have obtained high-quality sentence represen- tation based on contrastive learning from pre-trained models. However, these works suffer the inconsistency of input forms between the pre-training and fine-tuning stages. Also, they typically encode a sentence independently and lack feature interaction between sentences. To conquer these issues, we propose a novel Contrastive framework with Inter-sentence Interaction (Con- IsI), which introduces a sentence-level objective to improve sentence representation based on contrastive learning by fine-grained interaction between sentences. The sentence-level objective guides the model to focus on fine-grained semantic information by feature interaction between sentences, and we design three different sentence construction strategies to explore its effect. We conduct experiments on seven Semantic Textual Similarity (STS) tasks. The experimental results show that our ConIsI models based on BERTbase and RoBERTabase achieve state-of- the-art performance, substantially outperforming previous best models SimCSE-BERTbase and SimCSE-RoBERTabase by 2.05% and 0.77% respectively. 1 Introduction Learning good universal sentence representation is a fundamental task and benefits a wide range of nat- ural language processing tasks such as text classification and machine translation, especially for large- scale semantic similarity computation and information retrieval. With the rise of pre-trained language models (Devlin et al., 2019; Liu et al., 2019), many downstream tasks have achieved remarkable improve- ments. However, the native sentence representation derived from pre-trained language models without additional supervision are usually low-quality and can not be used directly (Reimers et al., 2019). Re- cently, contrastive learning has become a popular approach to improve the quality of sentence represen- tation in a self-supervised way. Contrastive learning is an approach of learning effective feature representation by positive pairs and negative pairs. It generally takes different views as positive or negative pairs for each sentence using var- ious data augmentation ways. And it works by pulling semantically close positive instances together and pushing negative instances away. However, current approaches based on contrastive learning mainly suf- fer two problems: train-tuned bias and fine-grained interaction deficiency. Firstly, previous approaches typically input a single sentence to the encoder at a time, which is inconsistent with the pre-training stage of the language models. Most language models concatenate multiple sentences as the input form at the pre-training stage. We argue that the inconsistency of input forms between the pre-training and fine-tuning stages may harm the performance. Secondly, each sentence in a minibatch is encoded inde- pendently while training, which lacks fine-grained interaction information between sentences. Accord- ing to previous works in text matching (Li et al., 2021; Wang et al., 2021; Lu et al., 2022) , modeling a proper interaction between input sentences can improve the performance of semantic feature embedding ©2022 China National Conference on Computational Linguistics Published under Creative Commons Attribution 4.0 International License ∗ : Corresponding Author CCL 2022 Proceedings of …

Content truncated to 500 words