DEPEVAL(summ): Dependency-based Evaluation for Automatic Summaries

Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pages 190–198, Suntec, Singapore, 2-7 August 2009. c⃝2009 ACL and AFNLP DEPEVAL(summ): Dependency-based Evaluation for Automatic Summaries Karolina Owczarzak Information Access Division National Institute of Standards and Technology Gaithersburg, MD 20899 karolina.owczarzak@nist.gov Abstract This paper presents DEPEVAL(summ), a dependency-based metric for automatic evaluation of summaries. Using a rerank- ing parser and a Lexical-Functional Gram- mar (LFG) annotation, we produce a set of dependency triples for each sum- mary. The dependency set for each candidate summary is then automatically compared against dependencies generated from model summaries. We examine a number of variations of the method, in- cluding the addition of WordNet, par- tial matching, or removing relation la- bels from the dependencies. In a test on TAC 2008 and DUC 2007 data, DE- PEVAL(summ) achieves comparable or higher correlations with human judg- ments than the popular evaluation metrics ROUGE and Basic Elements (BE). 1 Introduction Evaluation is a crucial component in the area of automatic summarization; it is used both to rank multiple participant systems in shared summariza- tion tasks, such as the Summarization track at Text Analysis Conference (TAC) 2008 and its Docu- ment Understanding Conference (DUC) predeces- sors, and to provide feedback to developers whose goal is to improve their summarization systems. However, manual evaluation of a large number of documents necessary for a relatively unbiased view is often unfeasible, especially in the contexts where repeated evaluations are needed. Therefore, there is a great need for reliable automatic metrics that can perform evaluation in a fast and consistent manner. In this paper, we explore one such evaluation metric, DEPEVAL(summ), based on the compar- ison of Lexical-Functional Grammar (LFG) de- pendencies between a candidate summary and one or more model (reference) summaries. The method is similar in nature to Basic Elements (Hovy et al., 2005), in that it extends beyond a simple string comparison of word sequences, reaching instead to a deeper linguistic analysis of the text. Both methods use hand-written ex- traction rules to derive dependencies from con- stituent parses produced by widely available Penn II Treebank parsers. The difference between DEPEVAL(summ) and BE is that in DEPE- VAL(summ) the dependency extraction is accom- plished through an LFG annotation of Cahill et al. (2004) applied to the output of the reranking parser of Charniak and Johnson (2005), whereas in BE (in the version presented here) dependen- cies are generated by the Minipar parser (Lin, 1995). Despite relying on a the same concept, our approach outperforms BE in most comparisons, and it often achieves higher correlations with hu- man judgments than the string-matching metric ROUGE (Lin, 2004). A more detailed description of BE and ROUGE is presented in Section 2, which also gives an ac- count of manual evaluation methods employed at TAC 2008. Section 3 gives a short introduction to the LFG annotation. Section 4 describes in more detail DEPEVAL(summ) and its variants. Sec- tion 5 presents the experiment in which we com- …

Content truncated to 500 words