TermITH-Eval: a French Standard-Based Resource for Keyphrase Extraction Evaluation

TermITH-Eval: a French Standard-Based Resource for Keyphrase Extraction Evaluation Adrien Bougouin1, Sabine Barreaux2, Laurent Romary3, Florian Boudin1, B´eatrice Daille1 1 Universit´e de Nantes, LINA, 2 rue de la Houssini`ere, 44322 Nantes, France 2 INIST-CNRS, 2, all´ee du parc de Brabois, 54519 Vandoeuvre-l`es-Nancy, France 3 Humboldt-Universit¨at zu Berlin, Dorotheenstraße 24, 10117 Berlin, Germany E-mail: adrien.bougouin@univ-nantes.fr, sabine.barreaux@inist.fr, laurent.romary@inria.fr, florian.boudin@univ-nantes.fr, beatrice.daille@univ-nantes.fr Abstract Keyphrase extraction is the task of finding phrases that represent the important content of a document. The main aim of keyphrase extraction is to propose textual units that represent the most important topics developed in a document. The output keyphrases of automatic keyphrase extraction methods for test documents are typically evaluated by comparing them to manually assigned reference keyphrases. Each output keyphrase is considered correct if it matches one of the reference keyphrases. However, the choice of the appropriate textual unit (keyphrase) for a topic is sometimes subjective and evaluating by exact matching underestimates the performance. This paper presents a dataset of evaluation scores assigned to automatically extracted keyphrases by human evaluators. Along with the reference keyphrases, the manual evaluations can be used to validate new evaluation measures. Indeed, an evaluation measure that is highly correlated to the manual evaluation is appropriate for the evaluation of automatic keyphrase extraction methods. Keywords: TermITH-Eval, structured resource, automatic evaluation, keyphrase extraction. 1. Introduction and Motivation Keyphrases are textual units (words and phrases) that rep- resent the most important topics of a document. Keyphrase extraction is the task of automatically detecting those top- ics in the content of a document. The common prac- tice to evaluate the performance of keyphrase extraction systems is to compute the number of exact matches be- tween extracted keyphrases and (human assigned) refer- ence keyphrases (Hasan and Ng, 2014). However, this leads to overly pessimistic scores since variations in the extracted keyphrases that might be judged as correct cannot be taken into account (Zesch and Gurevych, 2009). Producing a more reliable estimate of system performance is not an easy task as assessing whether a textual unit is a keyphrase is highly subjective (Kim et al., 2010). Yet, a handful of attempts have been made in this direction (Zesch and Gurevych, 2009; Kim et al., 2010) but with limited suc- cess. The initiating work of Zesch and Gurevych (2009) stated the need for partial matching instead of exact match- ing but did not show the effectiveness of their measure com- pared with a human evaluation. Kim et al. (2010) improved the measure of Zesch and Gurevych (2009) and evaluated the correlation of both the original and improved measures with human evaluations. Computing the correlation be- tween an automatic evaluation measure and human eval- uators is an effective way of measuring how close the auto- matic judgment is to human judgment. However, the results shown by Kim et al. (2010) were not significant enough to influence automatic evaluation in recent work. Also, Kim et al. (2010) did not provide the manual evaluation data they used to correlate the evaluation measures with the …

Content truncated to 500 words