Collecting Highly Parallel Data for Paraphrase Evaluation

Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 190–200, Portland, Oregon, June 19-24, 2011. c⃝2011 Association for Computational Linguistics Collecting Highly Parallel Data for Paraphrase Evaluation David L. Chen Department of Computer Science The University of Texas at Austin Austin, TX 78712, USA dlcc@cs.utexas.edu William B. Dolan Microsoft Research One Microsoft Way Redmond, WA 98052, USA billdol@microsoft.com Abstract A lack of standard datasets and evaluation metrics has prevented the field of paraphras- ing from making the kind of rapid progress enjoyed by the machine translation commu- nity over the last 15 years. We address both problems by presenting a novel data collection framework that produces highly parallel text data relatively inexpensively and on a large scale. The highly parallel nature of this data allows us to use simple n-gram comparisons to measure both the semantic adequacy and lex- ical dissimilarity of paraphrase candidates. In addition to being simple and efficient to com- pute, experiments show that these metrics cor- relate highly with human judgments. 1 Introduction Machine paraphrasing has many applications for natural language processing tasks, including ma- chine translation (MT), MT evaluation, summary evaluation, question answering, and natural lan- guage generation. However, a lack of standard datasets and automatic evaluation metrics has im- peded progress in the field. Without these resources, researchers have resorted to developing their own small, ad hoc datasets (Barzilay and McKeown, 2001; Shinyama et al., 2002; Barzilay and Lee, 2003; Quirk et al., 2004; Dolan et al., 2004), and have often relied on human judgments to evaluate their results (Barzilay and McKeown, 2001; Ibrahim et al., 2003; Bannard and Callison-Burch, 2005). Consequently, it is difficult to compare different sys- tems and assess the progress of the field as a whole. Despite the similarities between paraphrasing and translation, several major differences have prevented researchers from simply following standards that have been established for machine translation. Pro- fessional translators produce large volumes of bilin- gual data according to a more or less consistent spec- ification, indirectly fueling work on machine trans- lation algorithms. In contrast, there are no “profes- sional paraphrasers”, with the result that there are no readily available large corpora and no consistent standards for what constitutes a high-quality para- phrase. In addition to the lack of standard datasets for training and testing, there are also no standard metrics like BLEU (Papineni et al., 2002) for eval- uating paraphrase systems. Paraphrase evaluation is inherently difficult because the range of potential paraphrases for a given input is both large and unpre- dictable; in addition to being meaning-preserving, an ideal paraphrase must also diverge as sharply as possible in form from the original while still sound- ing natural and fluent. Our work introduces two novel contributions which combine to address the challenges posed by paraphrase evaluation. First, we describe a frame- work for easily and inexpensively crowdsourcing ar- bitrarily large training and test sets of independent, redundant linguistic descriptions of the same seman- tic content. Second, we define a new evaluation metric, …

Content truncated to 500 words