Approximate Matching for Evaluating Keyphrase Extraction

International Conference RANLP 2009 - Borovets, Bulgaria, pages 484–489 Approximate Matching for Evaluating Keyphrase Extraction Torsten Zesch and Iryna Gurevych Ubiquitous Knowledge Processing Lab Computer Science Department Technische Universit¨at Darmstadt, D-64289 Darmstadt, Germany http://www.ukp.tu-darmstadt.de Abstract We propose a new evaluation strategy for keyphrase extraction based on approximate keyphrase matching. It corresponds well with human judgments and is better suited to as- sess the performance of keyphrase extraction ap- proaches. Additionally, we propose a general- ized framework for comprehensive analysis of keyphrase extraction that subsumes most exist- ing approaches, which allows for fair testing con- ditions. For the first time, we compare the re- sults of state-of-the-art unsupervised and super- vised keyphrase extraction approaches on three evaluation datasets and show that the relative performance of the approaches heavily depends on the evaluation metric as well as on the prop- erties of the evaluation dataset. Keywords keyphrase extraction; approximate matching 1 Introduction Keyphrases are small sets of expressions representing a document’s content. Keyphrase extraction is the task of automatically extracting such keyphrases from a document. The extracted phrases have to be present in the document itself, in contrast to keyphrase assign- ment (a multi-class text classification problem) where a fixed set of keyphrases is used which are not necessar- ily contained in the document. Keyphrase extraction has important applications in NLP including summa- rization [4, 11], clustering [9], as well as indexing and browsing [8], highlighting [22] and searching [2]. Despite the importance of the task, the evaluation of keyphrase extraction has not received much research attention in the past. In this paper, we address three core problems with the evaluation of keyphrase ex- traction: (i) the evaluation metric, (ii) the evaluation datasets, and (iii) the evaluation framework. The performance of most keyphrase extraction al- gorithms is evaluated by comparing whether the ex- tracted keyphrases exactly match the human assigned gold standard keyphrases. However, this is known to underestimate performance [22]. Allowing only ex- act matchings cannot account for variations in the ex- tracted keyphrases that might be perfectly acceptable when presented to humans. For example, longer noun phrases like “congress party spokesman” are usually more specific and thus more informative to the reader than shorter noun phrases like “congress party”. How- ever, due to reading and writing economy, specific terms are usually not often repeated in a document [1]. Thus, longer noun phrases are unlikely to be annotated by human annotators, preventing exact matching. To compensate for these shortcomings, we propose a new approximate matching strategy that also accounts for non-exact matches and is able to give a better picture of the actual quality of a keyphrase extraction algo- rithm. We evaluate the validity of the new matching strategy in a human annotation study in Section 3. The lack of standard datasets is the second problem tackled in this paper. Comparing results from different papers is difficult as no standard datasets are used, and few papers have compared their results on more than one dataset with different competing systems. Thus it …

Content truncated to 500 words