Sentence Alignment for Monolingual Comparable Corpora

Sentence Alignment for Monolingual Comparable Corpora Regina Barzilay Department of Computer Science Cornell University Ithaca, NY 14853 regina@cs.cornell.edu Noemie Elhadad Department of Computer Science Columbia University New York, NY 10027 noemie@cs.columbia.edu Abstract We address the problem of sentence align- ment for monolingual corpora, a phe- nomenon distinct from alignment in par- allel corpora. Aligning large compara- ble corpora automatically would provide a valuable resource for learning of text-to- text rewriting rules. We incorporate con- text into the search for an optimal align- ment in two complementary ways: learn- ing rules for matching paragraphs using topic structure and further refining the matching through local alignment to find good sentence pairs. Evaluation shows that our alignment method outperforms state-of-the-art systems developed for the same task. 1 Introduction Text-to-text generation is an emerging area of re- search in NLP (Chandrasekar and Bangalore, 1997; Caroll et al., 1999; Knight and Marcu, 2000; Jing and McKeown, 2000). Unlike in traditional concept- to-text generation, text-to-text generation applica- tions take a text as input and transform it into a new text satisfying specific constraints, such as length in summarization or style in text simplification. One exciting new research direction is the automatic in- duction of such transformation rules. This is a par- ticularly promising direction given that there are nat- urally occurring examples of comparable texts that convey the same information yet are written in dif- ferent styles. Presented with two such texts, one can pair sentences that convey the same information, thereby building a training set of rewriting examples for the domain to which the texts belong. We believe that automating this process will provide researchers in text-to-text generation with valuable training and testing resources, just as techniques to align multi- lingual parallel corpora boosted research in Machine Translation (MT). In this paper, we address the task of aligning sen- tences in text pairs. We focus on monolingual com- parable corpora, that is, texts in the same language (e.g., English) that overlap in the information they convey. Stories about the same events from different press agencies and texts presenting the same infor- mation to experts and lay people are two examples. In MT, the task of sentence alignment was exten- sively studied for parallel corpora.1 A typical sen- tence alignment algorithm can be roughly described as a two-step process: (1) for each sentence pair compute a local similarity value, independently of the other sentences; (2) find an overall sequence of mapped sentences, using both the local similarity values and additional features. In the case of monolingual corpora, step (2) might seem unnecessary. Since the texts share the same language, it would be enough to choose for local similarity a function based on lexical cues only and select sentence pairs with high lexical similarity. Even a simple lexical function (e.g., one that counts word overlap) could produce an accurate alignment. 1Sentence alignment for comparable multilingual corpora was not addressed in previous research. Comparable cor- pora have primarily been used to build bilingual lexical re- sources (Fung and …

Content truncated to 500 words