A French Corpus for Semantic Similarity

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6889–6894 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6889 A French Corpus for Semantic Similarity R´emi Cardon, Natalia Grabar UMR 8163 STL, CNRS, Universit´e de Lille Domaine du Pont de bois 59653 Villeneuve d’Ascq CEDEX, France {remi.cardon, natalia.grabar}@univ-lille.fr Abstract Semantic similarity is an area of Natural Language Processing that is useful for several downstream applications, such as machine translation, natural language generation, information retrieval, or question answering. The task consists in assessing the extent to which two sentences express or do not express the same meaning. To do so, corpora with graded pairs of sentences are required. The grade is positioned on a given scale, usually going from 0 (completely unrelated) to 5 (equivalent semantics). In this work, we introduce such a corpus for French, the first that we know of. It is comprised of 1,010 sentence pairs with grades from five annotators. We describe the annotation process, analyse these data, and perform a few experiments for the automatic grading of semantic similarity. Keywords: semantic similarity, manual annotation, French language, regression 1. Introduction Semantic textual similarity is a subtask of Natural Lan- guage Processing. At the level of sentences, the task con- sists in evaluating to what extent two sentences express the same meaning. This task is useful for several applications, such as machine translation, text summarization, informa- tion retrieval, natural language generation, or text simplifi- cation (Wieting et al., 2019; Vadapalli et al., 2017; Yasui et al., 2019; Kajiwara and Komachi, 2016). The computing of the semantic textual similarity requires corpora with anno- tated pairs of sentences. The annotation is most of the time performed on a continuous scale where scores range from 0 (the sentences express completely unrelated meanings) to 5 (the meaning is exactly the same in both sentences). Several challenges dedicated to semantic textual similarity (STS) have been held within the SemEval evaluation cam- paign between 2012 and 2017. STS provides the research community with bilingual and monolingual data. In our work, we are interested in monolingual semantic similar- ity. In relation with the monolingual semantic similarity, data from a few languages (English, Spanish and Arabic) have been exploited (Cer et al., 2017) and made available for the research community. The overall STS benchmark data for English1, with data taken from editions held from 2012 to 2017, contains 8,628 sentence pairs, while only 250 sentence pairs were proposed for Spanish and for Ara- bic. Besides, similar data are also proposed for Portuguese through the ASSIN workshop (Feitosa and Pinheiro, 2017) dataset, which is composed of 10,000 pairs – 5,000 for Brazilian Portuguese and 5,000 for European Portuguese. All those datasets are taken from general language and vari- ous sources : news articles, forum posts and video subtitles. Yet, there is no similar data in French. In our work, we introduce a semantic textual similarity cor- pus for French. We first describe the data that have been used …

Content truncated to 500 words