Extrinsic Evaluation of French Dependency Parsers on a Specialized Corpus: Comparison of Distributional Thesauri

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5820–5828 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5820 Extrinsic Evaluation of French Dependency Parsers on a Specialized Corpus: Comparison of Distributional Thesauri Ludovic Tanguy1, Pauline Brunet2, Olivier Ferret2 1. CLLE-ERSS: CNRS & Université de Toulouse, France 2. CEA, LIST, F-91191 Gif-sur-Yvette, France ludovic.tanguy@univ-tlse2.fr {pauline.brunet,olivier.ferret}@cea.fr Abstract We present a study in which we compare 11 different French dependency parsers on a specialized corpus (consisting of research articles on NLP from the proceedings of the TALN conference). Due to the lack of a suitable gold standard, we use each of the parsers’ output to generate distributional thesauri using a frequency-based method. We compare these 11 thesauri to assess the impact of choosing a parser over another. We show that, without any reference data, we can still identify relevant subsets among the different parsers. We also show that the similarity we identify between parsers is confirmed on a restricted distributional benchmark. Keywords: French dependency parsing, distributional semantics, specialized corpus 1. Introduction This article takes place in the context of a more global re- search about distributional semantic analysis methods for specialized domains in French. Our medium-term objective is to select the most efficient method for identifying seman- tic similarities between the lexical and terminological units of a small specialized corpus (a few million words at most). This task is conditioned by several parameters that must be chosen carefully along the way from the extraction of word contexts to the construction of a distributional model. Unfortunately, these choices cannot be made on the sole basis of the knowledge acquired on the large and generic corpora that are commonly used to compare distributional models. Although this is no more a commonly used config- uration, we selected a method relying on syntactic contexts and based on the assumption that the small amount of data can be compensated to some extent by a rich linguistic an- notation. This has been partially demonstrated in (Tanguy et al., 2015). Our focus here is how the choice of the parser impacts the construction of a distributional model using a frequency-based method. This is related to the wider issue of comparing the efficiency of different available tools and models for dependency parsing. Many efforts have been made in the French Natural Lan- guage Processing (NLP) community to compare differ- ent parsers: the Easy (Paroubek et al., 2008), Passage (De La Clergerie et al., 2008), SPMRL (Seddah et al., 2013) and CoNLL (Zeman et al., 2018) shared tasks as well as more focused studies like (Candito et al., 2010) or (De La Clergerie, 2014). However, the benchmarks used in these tasks and studies, adopting the kind of diverse, generic corpora on which the tools have been trained, might not be the most relevant option for parsing specialized cor- pora. Moreover, even though some of these campaigns are recent, all the main tools currently available have not been evaluated on the same benchmarks. …

Content truncated to 500 words