What Makes Sentences Semantically Related? A Textual Relatedness Dataset and Empirical Study

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 782–796 May 2-6, 2023 ©2023 Association for Computational Linguistics What Makes Sentences Semantically Related? A Textual Relatedness Dataset and Empirical Study Mohamed Abdalla Institute for Better Health & University of Toronto msa@cs.toronto.edu Krishnapriya Vishnubhotla University of Toronto vkpriya@cs.toronto.edu Saif M. Mohammad National Research Council Canada saif.mohammad@nrc-cnrc.gc.ca Abstract The degree of semantic relatedness of two units of language has long been considered funda- mental to understanding meaning. Addition- ally, automatically determining relatedness has many applications such as question answering and summarization. However, prior NLP work has largely focused on semantic similarity, a subset of relatedness, because of a lack of relat- edness datasets. In this paper, we introduce a dataset for Semantic Textual Relatedness, STR- 2022, that has 5,500 English sentence pairs manually annotated using a comparative an- notation framework, resulting in fine-grained scores. We show that human intuition regarding relatedness of sentence pairs is highly reliable, with a repeat annotation correlation of 0.84. We use the dataset to explore questions on what makes sentences semantically related. We also show the utility of STR-2022 for evaluating automatic methods of sentence representation and for various downstream NLP tasks. Our dataset, data statement, and annotation questionnaire can be found at: https://doi. org/10.5281/zenodo.7599667. 1 Introduction The semantic relatedness of two units of language is the degree to which they are close in terms of their meaning (Mohammad, 2008; Mohammad and Hirst, 2012). The linguistic units can be words, phrases, sentences, etc. Though our intuition of semantic relatedness is dependent on many factors such as the context of assessment, age, and socio- economic status (Harispe et al., 2015), it is argued that a consensus can usually be reached for many pairs (Harispe et al., 2015). Consider the two sen- tence pairs in Table 1. Most speakers of English will agree that the sentences in the first pair are closer in meaning to one another than those in the second. When judging the semantic relatedness between two sentences, humans generally look for commonalities in meaning: whether they are on the Pair 1: a. There was a lemon tree next to the house. b. The boy enjoyed reading under the lemon tree. Pair 2: a. There was a lemon tree next to the house. b. The boy was an excellent football player. Table 1: Most people will agree that the sentences in pair 1 are more related than the sentences in pair 2. same topic, express the same view, originate from the same time period, one elaborates on (or follows from) the other, etc. The semantic relatedness of two units of lan- guage has long been considered fundamental to understanding meaning (Halliday and Hasan, 1976; Miller and Charles, 1991); given how difficult it has been to define meaning, a natural approach to get at the meaning of a unit is to determine how close it is to other units. Thus, unsurprisingly, auto- matically determining relatedness has many appli- cations such as …

Content truncated to 500 words