ParaMetric: An Automatic Evaluation Metric for Paraphrasing

Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 97–104 Manchester, August 2008 ParaMetric: An Automatic Evaluation Metric for Paraphrasing Chris Callison-Burch Center for Speech and Language Processing Johns Hopkins University 3400 N. Charles St. Baltimore, MD 21218 Trevor Cohn Mirella Lapata School of Informatics University of Edinburgh 2 Buccleuch Place Edinburgh, EH8 9LW Abstract We present ParaMetric, an automatic eval- uation metric for data-driven approaches to paraphrasing. ParaMetric provides an ob- jective measure of quality using a collec- tion of multiple translations whose para- phrases have been manually annotated. ParaMetric calculates precision and recall scores by comparing the paraphrases dis- covered by automatic paraphrasing tech- niques against gold standard alignments of words and phrases within equivalent sen- tences. We report scores for several estab- lished paraphrasing techniques. 1 Introduction Paraphrasing is useful in a variety of natural lan- guage processing applications including natural language generation, question answering, multi- document summarization and machine translation evaluation. These applications require paraphrases for a wide variety of domains and language us- age. Therefore building hand-crafted lexical re- sources such as WordNet (Miller, 1990) would be far too laborious. As such, a number of data-driven approaches to paraphrasing have been developed (Lin and Pantel, 2001; Barzilay and McKeown, 2001; Barzilay and Lee, 2003; Pang et al., 2003; Quirk et al., 2004; Bannard and Callison-Burch, 2005). Despite this spate of research, no objective evaluation metric has been proposed. In absence of a repeatable automatic evaluation, the quality of these paraphrasing techniques was gauged using subjective manual evaluations. Sec- tion 2 gives a survey of the various evaluation methodologies used in previous research. It has not been possible to directly compare paraphrasing c⃝ 2008. Licensed under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported li- cense (http://creativecommons.org/licenses/by-nc-sa/3.0/). Some rights reserved. techniques, because each one was evaluated using its own idiosyncratic experimental design. More- over, because these evaluations were performed manually, they are difficult to replicate. We introduce an automatic evaluation metric, called ParaMetric, which uses paraphrasing tech- niques to be compared and enables an evaluation to be easily repeated in subsequent research. Para- Metric utilizes data sets which have been annotated with paraphrases. ParaMetric compares automatic paraphrases against reference paraphrases. In this paper we: • Present a novel automatic evaluation metric for data-driven paraphrasing methods; • Describe how manual alignments are cre- ated by annotating correspondences between words in multiple translations; • Show how phrase extraction heuristics from statistical machine translation can be used to enumerate paraphrases from the alignments; • Report ParaMetric scores for a number of ex- isting paraphrasing methods. 2 Related Work No consensus has been reached with respect to the proper methodology to use when evaluating para- phrase quality. This section reviews past methods for paraphrase evaluation. Researchers usually present the quality of their automatic paraphrasing technique in terms of a subjective manual evaluation. These have used a variety of criteria. For example, Barzilay and McKeown (2001) evaluated their paraphrases by asking judges whether paraphrases were “ap- proximately conceptually equivalent.” …

Content truncated to 500 words