Comparison and Combination of Sentence Embeddings Derived from Different Supervision Signals

Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 139 - 150 July 14-15, 2022 ©2022 Association for Computational Linguistics Comparison and Combination of Sentence Embeddings Derived from Different Supervision Signals Hayato Tsukagoshi Ryohei Sasano Koichi Takeda Graduate School of Informatics, Nagoya University tsukagoshi.hayato.r2@s.mail.nagoya-u.ac.jp, {sasano,takedasu}@i.nagoya-u.ac.jp Abstract There have been many successful applications of sentence embedding methods. However, it has not been well understood what properties are captured in the resulting sentence embed- dings depending on the supervision signals. In this paper, we focus on two types of sentence embedding methods with similar architectures and tasks: one fine-tunes pre-trained language models on the natural language inference task, and the other fine-tunes pre-trained language models on word prediction task from its defini- tion sentence, and investigate their properties. Specifically, we compare their performances on semantic textual similarity (STS) tasks using STS datasets partitioned from two perspectives: 1) sentence source and 2) superficial similar- ity of the sentence pairs, and compare their performances on the downstream and probing tasks. Furthermore, we attempt to combine the two methods and demonstrate that combin- ing the two methods yields substantially better performance than the respective methods on unsupervised STS tasks and downstream tasks. 1 Introduction Sentence embeddings are dense vector representa- tions of a sentence. A variety of methods have been proposed to derive sentence embeddings, includ- ing those based on unsupervised learning (Kiros et al., 2015; Hill et al., 2016; Logeswaran and Lee, 2018; Cer et al., 2018; Wang et al., 2021) and super- vised learning (Conneau et al., 2017). Pre-trained Transformer-based (Vaswani et al., 2017) language models, such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), have been successfully applied in a wide range of NLP tasks, and sentence embedding methods that leverage pre-trained lan- guage models have also performed well on seman- tic textual similarity (STS) tasks and several down- stream tasks. These methods refine pre-trained language models for sophisticated sentence embed- dings by unsupervised learning (Li et al., 2020; Definition sentence Sentence B Sentence A w|V| w1 w2 w3 ... BERT BERT pooling pooling Contradiction Entailment Neutral Label prediction layer BERT pooling Word prediction layer Figure 1: Overviews of SBERT (left) and DefSent (right). Wang and Kuo, 2020; Giorgi et al., 2021; Carlsson et al., 2021; Yan et al., 2021; Gao et al., 2021), or supervised learning (Reimers and Gurevych, 2019; Tsukagoshi et al., 2021; Gao et al., 2021). Among them, Reimers and Gurevych (2019) pro- posed Sentence-BERT (SBERT), which fine-tunes pre-trained language models on the natural lan- guage inference (NLI) task. SBERT performed well on the STS and downstream tasks. Recently, Tsukagoshi et al. (2021) proposed DefSent, which fine-tunes pre-trained language models on the task of predicting a word from its definition sentence in a dictionary, and reported that it performed com- parably to SBERT. Figure 1 shows overviews of SBERT and DefSent. Although both methods fine- tune the same pre-trained models and use the same pooling operations to derive a sentence embedding, the …

Content truncated to 500 words