Evaluating N-gram based Evaluation Metrics for Automatic Keyphrase Extraction

Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 572–580, Beijing, August 2010 Evaluating N-gram based Evaluation Metrics for Automatic Keyphrase Extraction Su Nam Kim, Timothy Baldwin CSSE University of Melbourne sunamkim@gmail.com, tb@ldwin.net Min-Yen Kan School of Computing National University of Singapore kanmy@comp.nus.edu.sg Abstract This paper describes a feasibility study of n-gram-based evaluation metrics for automatic keyphrase extraction. To ac- count for near-misses currently ignored by standard evaluation metrics, we adapt various evaluation metrics developed for machine translation and summarization, and also the R-precision evaluation metric from keyphrase evaluation. In evaluation, the R-precision metric is found to achieve the highest correlation with human anno- tations. We also provide evidence that the degree of semantic similarity varies with the location of the partially-matching component words. 1 Introduction Keyphrases are noun phrases (NPs) that are repre- sentative of the main content of documents. Since they represent the key topics in documents, ex- tracting good keyphrases benefits various natu- ral language processing (NLP) applications such as summarization, information retrieval (IR) and question-answering (QA). Keyphrases can also be used in text summarization as semantic metadata (Barzilay and Elhadad, 1997; Lawrie et al., 2001; D’Avanzo and Magnini, 2005). In search engines, keyphrases supplement full-text indexing and as- sist users in creating good queries. In the past, a large body of work on keyphrases has been carried out as an extraction task, uti- lizing three types of cohesion: (1) document cohesion, i.e. cohesion between documents and keyphrases (Frank et al., 1999; Witten et al., 1999; Matsuo and Ishizuka, 2004; Medelyan and Wit- ten, 2006; Nguyen and Kan, 2007; Wan and Xiao, 2008); (2) keyphrase cohesion, i.e. cohe- sion among keyphrases (Turney, 2003); and (3) term cohesion, i.e. cohesion among terms in a keyphrase (Park et al., 2004). Despite recent successes in keyphrase extrac- tion (Frank et al., 1999; Turney, 2003; Park et al., 2004; Medelyan and Witten, 2006; Nguyen and Kan, 2007), current work is hampered by the in- flexibility of standard metrics in evaluating differ- ent approaches. As seen in other fields, e.g. ma- chine translation (MT) and multi-document sum- marization, the advent of standardized automatic evaluation metrics, combined with standardized datasets, has enabled easy comparison of sys- tems and catalyzed the respective research ar- eas. Traditionally, the evaluation of automatic keyphrase extraction has relied on the number of exact matches in author-assigned keyphrases and reader-assigned keyphrases. The main prob- lem with this approach is that even small vari- ants in the keyphrases are not given any credit. For example, given the gold-standard keyphrase effective grid computing algorithm, grid com- puting algorithm is a plausible keyphrase candi- date and should be scored appropriately, rather than being naively evaluated as wrong. Addition- ally, author-assigned keyphrases and even reader- assigned keyphrases often have their own prob- lems in this type of evaluation (Medelyan and Wit- ten, 2006). For example, some keyphrases are often partly or wholly subsumed by other can- didates or may not even occur in the document. Therefore, counting the exactly-matching candi- …

Content truncated to 500 words