Extending the gold standard for a lexical substitution task: is it worth it?

Extending the Gold Standard for a Lexical Substitution Task: is it worth it? Ludovic Tanguy (1), C´ecile Fabre (1) , Laura Rivi`ere (2) (1) CLLE-ERSS: CNRS & University of Toulouse, France (2) University of Toulouse, France ludovic.tanguy@univ-tlse2.fr, cecile.fabre@univ-tlse2.fr, laurariviere12@gmail.com Abstract We present a new evaluation scheme for the lexical substitution task. Following (McCarthy and Navigli, 2007) we conducted an annotation task for French that mixes two datasets: in the first one, 300 sentences containing a target word (among 30 different) were submitted to annotators who were asked to provide substitutes. The second one contains the propositions of the systems that participated to the lexical substitution task based on the same data. The idea is first, to assess the capacity of the systems to provide good substitutes that would not have been proposed by the annotators and second, to measure the impact on the task evaluation of a new gold standard that incorporates these additional data. While (McCarthy and Navigli, 2009) have conducted a similar post hoc analysis, re-evaluation of the systems’ performances has not been carried out to our knowledge. This experiment shows interesting differences between the two resulting datasets and gives insight on how automatically retrieved substitutes can provide complementary data to a lexical production task, without however a major impact on the evaluation of the systems. Keywords: Lexical substitution, Gold Standard, Post hoc 1. Introduction The lexical substitution task consists in providing the best substitute for a given word in a given context, usually the sentence. This is a crucial task in Natural Language Pro- cessing that requires systems to identify words that are se- mantically close to the target, and to select among candi- dates those that best fit the context. The task has been intro- duced in SemEval-2007 by (McCarthy and Navigli, 2007) and involves annotators that are asked to provide substitutes for a single target word in context. This task has been re- produced with variations on the language (Cholakov et al., 2014; Fabre et al., 2014) or the size of the dataset (Kremer et al., 2014). In this paper, we estimate the reliability of this setup by per- forming an additional annotation task for French, based on the results of the systems. We follow and extend (McCarthy and Navigli, 2009) who conducted a post hoc analysis to evaluate the reliability of the gold standard. The idea is to assess the capacity of the systems to provide good substi- tutes that would not have been proposed by the annotators. This question is particularly relevant for the lexical sub- stitution task for two main reasons: first, finding a lexical substitute for a target word in the context of a sentence is a fairly difficult production task; second, systems that rely on huge corpora and lexical resources to detect semantic equivalents are likely to provide supplementary candidates that may be mistakenly ruled out by the evaluation proce- dure. Following (McCarthy and Navigli, 2009) we created a new annotation task mixing man-made and automatically- retrieved substitutes. But whereas …

Content truncated to 500 words