A French Version of the FraCaS Test Suite

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5887–5895 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5887 A French Version of the FraCaS Test Suite Maxime Amblard, Clément Beysson, Philippe de Groote, Bruno Guillaume, Sylvain Pogodalla Université de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France {maxime.amblard, clement.beysson}@univ-lorraine.fr {philippe.degroote, bruno.guillaume, sylvain.pogodalla}@inria.fr Abstract This paper presents a French version of the FraCaS test suite. This test suite, originally written in English, contains problems illustrating semantic inference in natural language. We describe linguistic choices we had to make when translating the FraCaS test suite in French, and discuss some of the issues that were raised by the translation. We also report an experiment we ran in order to test both the translation and the logical semantics underlying the problems of the test suite. This provides a way of checking formal semanticists’ hypotheses against actual semantic capacity of speakers (in the present case, French speakers), and allow us to compare the results we obtained with the ones of similar experiments that have been conducted for other languages. Keywords: semantics, inference, French 1. Introduction FraCaS (A Framework for Computational Semantics) was a European project on language research and engineering that ran from 1994 to 1996. It identified semantic inference as a significant part of computational semantic processing and developed an inference test suite to provide a way to evalu- ate and compare inferential competence of natural language processing systems and semantic theories, and proposed to test NLP system’s semantic capacity against inferenc- ing tasks. The proposed tests take the form that was popu- larized by textual entailment recognition (TER) tasks (Da- gan et al., 2006; Korman et al., 2018): the system is given an input text T together with a claim (a.k.a., hypothesis) H, and is “asked to determine whether [H] follows from T" (Cooper et al., 1996) (or, in Dagan et al. (2006)’s words, “if, typically, a human reading T would infer that H is most likely true”). Cooper et al. (1996, chap. 3) provided 346 problems ex- pressed in English and that are grouped by linguistic and semantic phenomena, e.g., monotonicity of quantifiers, col- lective and distributives plurals, ellipsis and anphora, etc. Figure 1. shows an example of the provided inferences: sentences above the line are the text T, the hypothesis H to be tested is worded as the question under the line, and the expected answer to the question is stated at the very bottom. Section 2. more precisely describes the range of problems and, in particular, the range of the possible ex- pected answer. Contrary to more recent TER tasks, lan- guage data were artificially build, in order to precisely il- lustrate semantic phenomena and to cover a wide range of natural language inference occurrences. (T) Every Italian man wants to be a great tenor. (T) Some Italian men are great tenors. (H) Are there Italian men who want to be a great tenor? [Yes] Figure 1: Problem 2 as …

Content truncated to 500 words