Aligning Sentences in Parallel Corpora

ALIGNING SENTENCES IN PARALLEL CORPORA Peter F. Brown, Jennifer C. Lai, a, nd Robert L. Mercer IBM Thomas J. Watson Research Center P.O. Box 704 Yorktown Heights, NY 10598 ABSTRACT In this paper we describe a statistical tech- nique for aligning sentences with their translations in two parallel corpora. In addition to certain anchor points that are available in our da.ta, the only information about the sentences that we use for calculating alignments is the number of tokens that they contain. Because we make no use of the lexical details of the sentence, the alignment com- putation is fast and therefore practical for appli- cation to very large collections of text. We have used this technique to align several million sen- tences in the English-French Hans~trd corpora and have achieved an accuracy in excess of 99% in a random selected set of 1000 sentence pairs that we checked by hand. We show that even without the benefit of anchor points the correlation between the lengths of aligned sentences is strong enough that we should expect to achieve an accuracy of between 96% and 97%. Thus, the technique may be applicable to a wider variety of texts than we have yet tried. INTRODUCTION Recent work by Brown et al., [Brown et al., 1988, Brown et al., 1990] has quickened anew the long dormant idea of using statistical techniques to carry out machine translation from one natural language to another. The lynchpin of their approach is a. large collection of pairs of sentences that. are mutual transla- tions. Beyond providing grist to the sta.tisti- cal mill, such pairs of sentences are valuable to researchers in bilingual lexicography [I(la.- va.ns and Tzoukerma.nn, 1990, Warwick and Russell, 1990] and may be usefifl in other ap- proaches to machine translation [Sadler, 1989]. In this paper, we consider the problem of extra.cting from pa.raJlel French and F, nglish corpora pairs sentences that are translations of one another. The task is not trivial because at times a single sentence in one language is translated as two or more sentences in the other language. At other times a sentence, or even a whole passage, may be missing from one or the other of the corpora. If a person is given two parallel texts and asked to match up the sentences in them, it is na.tural for him to look at the words in the sen- tences. Elaborating this intuitively appealing insight, researchers at Xerox and at ISSCO [Kay, 1991, Catizone et al., 1989] have devel- oped alignment Mgodthms that pair sentences according to the words that they contain. Any such algorithm is necessarily slow and, despite the potential for highly accurate alignment, may be unsuitable for very large collections of text. Our algorithm makes no use of the lexical details of the corpora, but deals only with the number of words in each sentence. Although we have used it only to align paral- lel French and English corpora from the pro- ceedings of the Canadian Parliament, we ex- …

Content truncated to 500 words