Unsupervised Construction of Large Paraphrase Corpora: Exploiting Massively Parallel News Sources

Unsupervised Construction of Large Paraphrase Corpora: Exploiting Massively Parallel News Sources Bill DOLAN, Chris QUIRK, and Chris BROCKETT Natural Language Processing Group, Microsoft Research One Microsoft Way Redmond, WA 90852, USA {billdol,chrisq,chrisbkt}@microsoft.com Abstract We investigate unsupervised techniques for acquiring monolingual sentence-level paraphrases from a corpus of temporally and topically clustered news articles collected from thousands of web-based news sources. Two techniques are employed: (1) simple string edit distance, and (2) a heuristic strategy that pairs initial (presumably summary) sentences from different news stories in the same cluster. We evaluate both datasets using a word alignment algorithm and a metric borrowed from machine translation. Results show that edit distance data is cleaner and more easily-aligned than the heuristic data, with an overall alignment error rate (AER) of 11.58% on a similarly-extracted test set. On test data extracted by the heuristic strategy, however, performance of the two training sets is similar, with AERs of 13.2% and 14.7% respectively. Analysis of 100 pairs of sentences from each set reveals that the edit distance data lacks many of the complex lexical and syntactic alternations that characterize monolingual paraphrase. The summary sentences, while less readily alignable, retain more of the non-trivial alternations that are of greatest interest learning paraphrase relationships. 1 Introduction The importance of learning to manipulate monolingual paraphrase relationships for applications like summarization, search, and dialog has been highlighted by a number of recent efforts (Barzilay & McKeown 2001; Shinyama et al. 2002; Lee & Barzilay 2003; Lin & Pantel 2001). While several different learning methods have been applied to this problem, all share a need for large amounts of data in the form of pairs or sets of strings that are likely to exhibit lexical and/or structural paraphrase alternations. One approach1 1 An alternative approach involves identifying anchor points--pairs of words linked in a known way--and collecting the strings that intervene. (Shinyama, et al. 2002; Lin & Pantel 2001). Since our interest is in that has been successfully used is edit distance, a measure of similarity between strings. The assumption is that strings separated by a small edit distance will tend to be similar in meaning: The leading indicators measure the economy… The leading index measures the economy…. Lee & Barzilay (2003), for example, use Multi- Sequence Alignment (MSA) to build a corpus of paraphrases involving terrorist acts. Their goal is to extract sentential templates that can be used in high-precision generation of paraphrase alter- nations within a limited domain. Our goal here is rather different: our interest lies in constructing a monolingual broad-domain corpus of pairwise aligned sentences. Such data would be amenable to conventional statistical machine translation (SMT) techniques (e.g., those discussed in Och & Ney 2003).2 In what follows we compare two strategies for unsupervised construction of such a corpus, one employing string similarity and the other associating sentences that may overlap very little at the string level. We measure the relative utility of the two derived monolingual corpora in the context of word alignment techniques developed originally for …

Content truncated to 500 words