ParaEval: Using Paraphrases to Evaluate Summaries Automatically

Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL, pages 447–454, New York, June 2006. c⃝2006 Association for Computational Linguistics ParaEval: Using Paraphrases to Evaluate Summaries Automatically Liang Zhou, Chin-Yew Lin, Dragos Stefan Munteanu, and Eduard Hovy University of Southern California Information Sciences Institute 4676 Admiralty Way Marina del Rey, CA 90292-6695 {liangz, cyl, dragos, hovy} @isi.edu Abstract ParaEval is an automated evaluation method for comparing reference and peer summaries. It facilitates a tiered- comparison strategy where recall-oriented global optimal and local greedy searches for paraphrase matching are enabled in the top tiers. We utilize a domain- independent paraphrase table extracted from a large bilingual parallel corpus us- ing methods from Machine Translation (MT). We show that the quality of ParaE- val’s evaluations, measured by correlating with human judgments, closely resembles that of ROUGE’s. 1 Introduction Content coverage is commonly measured in sum- mary comparison to assess how much information from the reference summary is included in a peer summary. Both manual and automatic methodolo- gies have been used. Naturally, there is a great amount of confidence in manual evaluation since humans can infer, paraphrase, and use world knowledge to relate text units with similar mean- ings, but which are worded differently. Human efforts are preferred if the evaluation task is easily conducted and managed, and does not need to be performed repeatedly. However, when resources are limited, automated evaluation methods become more desirable. For years, the summarization community has been actively seeking an automatic evaluation methodology that can be readily applied to various summarization tasks. ROUGE (Lin and Hovy, 2003) has gained popularity due to its simplicity and high correlation with human judgments. Even though validated by high correlations with human judgments gathered from previous Document Un- derstanding Conference (DUC) experiments, cur- rent automatic procedures (Lin and Hovy, 2003; Hovy et al., 2005) only employ lexical n-gram matching. The lack of support for word or phrase matching that stretches beyond strict lexical matches has limited the expressiveness and utility of these methods. We need a mechanism that sup- plements literal matching—i.e. paraphrase and synonym—and approximates semantic closeness. In this paper we present ParaEval, an automatic summarization evaluation method, which facili- tates paraphrase matching in an overall three-level comparison strategy. At the top level, favoring higher coverage in reference, we perform an opti- mal search via dynamic programming to find multi-word to multi-word paraphrase matches be- tween phrases in the reference summary (usually human-written) and those in the peer summary (system-generated). The non-matching fragments from the previous level are then searched by a greedy algorithm to find single-word para- phrase/synonym matches. At the third and the low- est level, we perform literal lexical unigram matching on the remaining texts. This tiered design for summary comparison guarantees at least a ROUGE-1 level of summary content matching if no paraphrases are found. The first two levels employ a paraphrase table. Since manually created multi-word paraphrases- —phrases determined by humans to be paraphrases of one another—are not …

Content truncated to 500 words