Cross-Level Semantic Similarity for Serbian Newswire Texts

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1691–1699 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1691 Cross-Level Semantic Similarity for Serbian Newswire Texts Vuk Batanović *, Maja Miličević Petrović † * Innovation Center of the School of Electrical Engineering, University of Belgrade, Bulevar kralja Aleksandra 73, 11120 Belgrade, Serbia † Department of Interpreting and Translation, University of Bologna, Corso della Repubblica 136, 47121 Forlì, Italy vuk.batanovic@ic.etf.bg.ac.rs, maja.milicevic2@unibo.it Abstract Cross-Level Semantic Similarity (CLSS) is a measure of the level of semantic overlap between texts of different lengths. Although this problem was formulated almost a decade ago, research on it has been sparse, and limited exclusively to the English language. In this paper, we present the first CLSS dataset in another language, in the form of CLSS.news.sr – a corpus of 1000 phrase-sentence and 1000 sentence-paragraph newswire text pairs in Serbian, manually annotated with fine-grained semantic similarity scores using a 0–4 similarity scale. We describe the methodology of data collection and annotation, and compare the resulting corpus to its preexisting counterpart in English, SemEval CLSS, following up with a preliminary linguistic analysis of the newly created dataset. State-of-the-art pre-trained language models are then fine-tuned and evaluated on the CLSS task in Serbian using the produced data, and their settings and results are discussed. The CLSS.news.sr corpus and the guidelines used in its creation are made publicly available. Keywords: cross-level semantic similarity, semantic textual similarity, corpus annotation 1. Introduction and Related Work Semantic similarity refers to the extent to which the meanings of two given text items are similar to each other. The level of semantic similarity is commonly expressed as a numerical score on a Likert scale. Establishing such similarity measurements is an integral part of various Natural Language Processing (NLP) tasks, such as Information Retrieval (Hliaoutakis et al., 2006), Question Answering (Risch et al., 2021), Text Summarization (Mnasri, de Chalendar, and Ferret, 2017), etc. Semantic similarity tasks typically focus on texts of similar length, such as individual words (Rubenstein and Goodenough, 1965), word senses (Budanitsky and Hirst, 2006), or sentences (Li et al., 2006). A well-known task of this sort is Semantic Textual Similarity (STS) (Corley and Mihalcea, 2005; Mihalcea, Corley, and Strapparava, 2006; Islam and Inkpen, 2008), popularized via a series of SemEval shared tasks (Agirre et al., 2012, 2013, 2014, 2015, 2016; Cer et al., 2017). In STS, the inputs being compared are short texts, usually the length of a sentence. By contrast, in Cross-Level Semantic Similarity (CLSS) the goal is to evaluate texts of different lengths, for example to semantically compare a phrase to a sentence, or a sentence to an entire paragraph. Naturally, this task has an additional level of complexity compared to its STS counterpart, since the length discrepancy all but ensures that the longer text will carry a greater amount of salient information than the shorter one. In effect, CLSS aims to measure how well the meaning of the longer text is summarized …

Content truncated to 500 words