Fine-grained Semantic Textual Similarity for Serbian

Fine-grained Semantic Textual Similarity for Serbian Vuk Batanović, Miloš Cvetanović, Boško Nikolić School of Electrical Engineering, University of Belgrade Bulevar kralja Aleksandra 73, 11120 Belgrade, Serbia vuk.batanovic@student.etf.bg.ac.rs, cmilos@etf.bg.ac.rs, nbosko@etf.bg.ac.rs Abstract Although the task of semantic textual similarity (STS) has gained in prominence in the last few years, annotated STS datasets for model training and evaluation, particularly those with fine-grained similarity scores, remain scarce for languages other than English, and practically non-existent for minor ones. In this paper, we present the Serbian Semantic Textual Similarity News Corpus (STS.news.sr) – an STS dataset for Serbian that contains 1192 sentence pairs annotated with fine-grained semantic similarity scores. We describe the process of its creation and annotation, and we analyze and compare our corpus with the existing news-based STS datasets in English and other major languages. Several existing STS models are evaluated on the Serbian STS News Corpus, and a new supervised bag-of-words model that combines part-of-speech weighting with term frequency weighting is proposed and shown to outperform similar methods. Since Serbian is a morphologically rich language, the effect of various morphological normalization tools on STS model performances is considered as well. The Serbian STS News Corpus, the annotation tool and guidelines used in its creation, and the STS model framework used in the evaluation are all made publicly available. Keywords: short-text semantic similarity, corpus annotation, morphological normalization 1. Introduction Semantic Textual Similarity (STS), sometimes also referred to as Short-text Semantic Similarity (STSS), is the task of assigning a numerical score to a given pair of short texts based on the level of semantic equivalence between them. The minimal numerical score in a given range indicates complete semantic independence, while the maximal score indicates full semantic equality. Although STS has important implications for a whole range of other natural language processing tasks, including information retrieval, question answering, machine translation, textual entailment, etc., research on this topic started appearing only around a decade ago (Corley and Mihalcea, 2005; Mihalcea, Corley, and Strapparava, 2006; Islam and Inkpen, 2008). Semantic Textual Similarity has gained in prominence since 2012, with its inclusion in the annual SemEval shared tasks (Agirre et al., 2012, 2013, 2014, 2015, 2016; Cer et al., 2017). A large collection of datasets with fine-grained semantic similarity scores has been annotated in this series of shared tasks, using a standardized methodology, and has been made publicly available. The move from binary similarity scores (Dolan and Brockett, 2005) to fine-grained ones has allowed for more precise model training and evaluation. However, most of this development has been limited to English. Several other major languages have been considered recently, including Spanish (Agirre et al., 2014, 2015; Cer et al., 2017), French (Vu et al., 2014), Portuguese (Fonseca et al., 2016), Chinese and Japanese (Hayashi and Luo, 2016), Arabic (Cer et al., 2017), and Hindi (Agarwal et al., 2017). Among them, only the datasets in Spanish, Portuguese, and Arabic have been made publicly available. To the best of our knowledge, there has been no development of STS datasets with …

Content truncated to 500 words