PARADIGM: Paraphrase Diagnostics through Grammar Matching

Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 192–201, Gothenburg, Sweden, April 26-30 2014. c⃝2014 Association for Computational Linguistics PARADIGM: Paraphrase Diagnostics through Grammar Matching Jonathan Weese and Juri Ganitkevitch Johns Hopkins University Chris Callison-Burch University of Pennsylvania Abstract Paraphrase evaluation is typically done ei- ther manually or through indirect, task- based evaluation. We introduce an in- trinsic evaluation PARADIGM which mea- sures the goodness of paraphrase col- lections that are represented using syn- chronous grammars. We formulate two measures that evaluate these paraphrase grammars using gold standard sentential paraphrases drawn from a monolingual parallel corpus. The first measure calcu- lates how often a paraphrase grammar is able to synchronously parse the sentence pairs in the corpus. The second mea- sure enumerates paraphrase rules from the monolingual parallel corpus and calculates the overlap between this reference para- phrase collection and the paraphrase re- source being evaluated. We demonstrate the use of these evaluation metrics on para- phrase collections derived from three dif- ferent data types: multiple translations of classic French novels, comparable sen- tence pairs drawn from different newspa- pers, and bilingual parallel corpora. We show that PARADIGM correlates with hu- man judgments more strongly than BLEU on a task-based evaluation of paraphrase quality. 1 Introduction Paraphrases are useful in a wide range of natu- ral language processing applications. A variety of data-driven approaches have been proposed to generate paraphrase resources (see Madnani and Dorr (2010) for a survey of these methods). Few objective metrics have been established to evalu- ate these resources. Instead, paraphrases are typi- cally evaluated using subjective manual evaluation or through task-based evaluations. Different researchers have used different crite- ria for manual evaluations. For example, Barzilay and McKeown (2001) evaluated their paraphrases by asking judges whether paraphrases were “ap- proximately conceptually equivalent.” Ibrahim et al. (2003) asked judges whether their para- phrases were “roughly interchangeable given the genre.” Bannard and Callison-Burch (2005) re- placed phrases with paraphrases in a number of sentences and asked judges whether the substitu- tions “preserved meaning and remained grammat- ical.” The results of these subjective evaluations are not easily reusable. Other researchers have evaluated their para- phrases through task-based evaluations. Lin and Pantel (2001) measured their potential impact on question-answering. Cohn and Lapata (2007) evaluate their applicability in the text-to-text gen- eration task of sentence compression. Zhao et al. (2009) use them to perform sentence compression and simplification and to compute sentence simi- larity. Several researchers have demonstrated that paraphrases can improve machine translation eval- uation (c.f. Kauchak and Barzilay (2006), Zhou et al. (2006), Madnani (2010) and Snover et al. (2010)). We introduce an automatic evaluation met- ric called PARADIGM, PARAphrase DIagnostics through Grammar Matching. This metric eval- uates paraphrase collections that are represented using synchronous grammars. Synchronous tree- adjoining grammars (STAGs), synchronous tree substitution grammars (STSGs), and synchronous context free grammars (SCFGs) are popular for- malisms for representing paraphrase rules (Dras, 1997; Cohn and Lapata, 2007; Madnani, 2010; Ganitkevitch et al., …

Content truncated to 500 words