SentBench: Comprehensive Evaluation of Self-Supervised Sentence Representation with Benchmark Construction

SentBench: Comprehensive Evaluation of Self-Supervised Sentence Representation with Benchmark Construction Xiaoming Liu1,3, Hongyu Lin1∗, Xianpei Han1,2∗, Le Sun1,2 1Chinese Information Processing Laboratory 2State Key Laboratory of Computer Science Institute of Software, Chinese Academy of Sciences, Beijing, China 3University of Chinese Academy of Sciences, Beijing, China {xiaoming2021,hongyu,xianpei,sunle}@iscas.ac.cn Abstract Self-supervised learning has been widely used to learn effective sentence representations. Previ- ous evaluation of sentence representations mainly focuses on the limited combination of tasks and paradigms while failing to evaluate their effectiveness in a wider range of application scenarios. Such divergences prevent us from understanding the limitations of current sentence representa- tions, as well as the connections between learning approaches and downstream applications. In this paper, we propose SentBench, a new comprehensive benchmark to evaluate sentence repre- sentations. SentBench covers 12 kinds of tasks and evaluates sentence representations with three types of different downstream application paradigms. Based on SentBench, we re-evaluate sev- eral frequently used self-supervised sentence representation learning approaches. Experiments show that SentBench can effectively evaluate sentence representations from multiple perspec- tives, and the performance on SentBench leads to some novel findings which enlighten future researches. 1 Introduction Self-supervised representation learning is considered an important reason for breakthroughs in NLP (Mikolov et al., 2013; Pennington et al., 2014; Peters et al., 2018; Devlin et al., 2019). And learn- ing effective sentence representations has long been a fundamental challenge. (Kiros et al., 2015; Conneau et al., 2017; Cer et al., 2018). In recent years, various self-supervised sentence representation learning approaches leverage different self-constrained signals, e.g., sentence pairs in the same narratives (Devlin et al., 2019), sentence order (Lan et al., 2019), or sentence permutation (Lewis et al., 2020), to learn representations by training models to distinguish positive instances from negatives. Even though current self-supervised sentence representation approaches have reached signifi- cant progress on some datasets like Semantic Textual Similarity (STS) (Ho and Nvasconcelos, 2020; Gao et al., 2021), benchmarks for evaluation lag far behind the development of methods (Wang et al., 2022). Currently, sentence representations are evaluated in limited tasks and specific paradigms. For example, the most commonly used SentEval benchmark (Conneau and Kiela, 2018) mainly focuses on single sentence classification and semantic similarity tasks. Unfortunately, prior lit- erature shows that performance on STS cannot reflect the effectiveness of sentence representations on a wider range of tasks (Reimers et al., 2016; Zhelezniak et al., 2019; Wang et al., 2022). And available evaluation toolkits assess the same downstream task with a singular paradigm, limiting our perception of methods in different application scenarios. Moreover, current self-supervised sentence representa- tion learning approaches are coupled with multiple factors, including diverse contrastive signals, training losses, and model architectures. Consequently, evaluating whether, where, and how a learning method will benefit the downstream tasks is difficult. In this paper, we propose SentBench, a new benchmark to comprehensively evaluate sentence repre- sentations with various downstream tasks and evaluation paradigms. As shown in Figure 1, SentBench ∗Corresponding authors. ©2023 China National Conference on Computational Linguistics Published under Creative Commons Attribution 4.0 …

Content truncated to 500 words