Sentence Similarity Based on Contexts

Sentence Similarity Based on Contexts Xiaofei Sun♦, Yuxian Meng♣, Xiang Ao▲, Fei Wu♦, Tianwei Zhang♥, Jiwei Li♦♣, and Chun Fan♠ ♦Zhejiang University, China, ♣Shannon.AI, China, ▲Chinese Academy of Sciences, China, ♥Nanyang Technological University, Singapore, ♠Computer Center, Peking University, China, ♠National Biomedical Imaging Center, Peking University, China, ♠Peng Cheng Laboratory, China {xiaofei sun,yuxian meng,jiwei li}@shannonai.com,aoxiang@ict.ac.cn wufei@zju.edu.cn,tianwei.zhang@ntu.edu.sg,fanchun@pku.edu.cn Abstract Existing methods to measure sentence similar- ity are faced with two challenges: (1) labeled datasets are usually limited in size, making them insufficient to train supervised neural models; and (2) there is a training-test gap for unsupervised language modeling (LM) based models to compute semantic scores between sentences, since sentence-level semantics are not explicitly modeled at training. This re- sults in inferior performances in this task. In this work, we propose a new framework to address these two issues. The proposed framework is based on the core idea that the meaning of a sentence should be defined by its contexts, and that sentence similarity can be measured by comparing the probabilities of generating two sentences given the same context. The proposed framework is able to generate high-quality, large-scale dataset with semantic similarity scores between two sen- tences in an unsupervised manner, with which the train-test gap can be largely bridged. Ex- tensive experiments show that the proposed framework achieves significant performance boosts over existing baselines under both the supervised and unsupervised settings across different datasets. 1 Introduction Measuring sentence similarity is a long-standing task in NLP (Luhn, 1957; Robertson et al., 1995; Blei et al., 2003; Peng et al., 2020). The task aims at quantitatively measuring the semantic re- latedness between two sentences, and has wide applications in text search (Farouk et al., 2018), natural language understanding (MacCartney and Manning, 2009), and machine translation (Yang et al., 2019a). One of the greatest challenges that existing methods face for sentence similarity is the lack of large-scale labeled datasets, which contain sentence pairs with labeled semantic similar- ity scores. The acquisition of such a dataset is both labor-intensive and expensive. For exam- ple, the STS benchmark (Cer et al., 2017) and SICK-Relatedness dataset (Marelli et al., 2014) respectively contain 8.6K and 9.8K labeled sen- tence pairs, the sizes of which are usually insuf- ficient for training deep neural networks. Unsupervised learning methods are proposed to address this issue, where word embeddings (Le and Mikolov, 2014) or BERT embeddings (Devlin et al., 2018) are used to to map sentences to fix-length vectors in an unsupervised manner. Then sentence similarity is computed based on the cosine or dot product of these sentence representa- tions. Our work follows this thread where sentence similarity is computed based on fix-length sen- tence representations, as opposed to comparing sentences directly. The biggest issue with cur- rent unsupervised approaches is that there exists a big gap between model training and testing (i.e., computing semantic similarity between two sentences). For example, the BERT-style mod- els are trained at the token level by predicting words given contexts, and there is neither explicit modeling sentence semantics …

Content truncated to 500 words