CLISTER : A Corpus for Semantic Textual Similarity in French Clinical Narratives

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4306–4315 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4306 CLISTER: A Corpus for Semantic Textual Similarity in French Clinical Narratives Nicolas Hiebel†, Olivier Ferret‡, Kar¨en Fort⋆, Aur´elie N´ev´eol† †Universit´e Paris Saclay, CNRS, LISN, France ‡Universit´e Paris-Saclay, CEA, List, F-91120, Palaiseau, France ⋆Sorbonne Universit´e / Universit´e de Lorraine, CNRS, Inria, LORIA, France †firstname.lastname@lisn.upsaclay.fr, ‡olivier.ferret@cea.fr, ⋆karen.fort@loria.fr Abstract Modern Natural Language Processing relies on the availability of annotated corpora for training and evaluating models. Such resources are scarce, especially for specialized domains in languages other than English. In particular, there are very few resources for semantic similarity in the clinical domain in French. This can be useful for many biomedical natural language processing applications, including text generation. We introduce a definition of similarity that is guided by clinical facts and apply it to the development of a new French corpus of 1,000 sentence pairs manually annotated according to similarity scores. This new sentence similarity corpus is made freely available to the community. We further evaluate the corpus through experiments of automatic similarity measurement. We show that a model of sentence embeddings can capture similarity with state-of-the-art performance on the DEFT STS shared task evaluation data set (Spearman=0.8343). We also show that the CLISTER corpus is complementary to DEFT STS. Keywords: Semantic Similarity, Corpus Development, Clinical Text, French 1. Introduction Semantic Textual Similarity (STS) is a Natural Lan- guage Processing (NLP) task aiming at evaluating the proximity between two pieces of text. It has various applications, such as question answering or text summarization, and has its roots in earlier work about paraphrase (Dolan et al., 2004) and textual en- tailment (Dagan et al., 2010). It is a well-known NLP task that has been regularly studied through evaluation campaigns like SemEval since 2007 and challenges like the 2020 edition of the French chal- lenge DEFT (D´efi Fouilles de Textes) (Cardon et al., 2020). Work on the automatic detection of sen- tence similarity is supported by resources such as the STS Benchmark dataset (Cer et al., 2017) and the SICK dataset (Marelli et al., 2014), which include pairs of sentences annotated with a degree of simi- larity (STS Benchmark) or a relation label (SICK dataset). To our knowledge, there are no existing STS-oriented corpora in French except the French Corpus for Semantic Similarity (Cardon and Grabar, 2020) from the DEFT challenge we men- tioned, and no STS corpora in the medical domain. Moreover, the notion of similarity is hard to define and existing corpora are not necessarily accompanied with guidelines providing an exact definition of the notion. In English, several STS corpora in the medical domain exist. The BIOSSES corpus (So˘gancıo˘glu et al., 2017) contains 100 sentence pairs from the Text Analysis Conference track on biomedical summarization in En- glish. Criteria for similarity are defined in this corpus but rely on the knowledge of the annotators for iden- tifying information that can be considered as ”impor- tant” …

Content truncated to 500 words