Aligning Noisy Parallel Corpora Across Language Groups: Word Pair Feature Matching by Dynamic Time Warping

Aligning Noisy Parallel Corpora Across Language Groups : Word Pair Feature Matching by Dynamic Time Warping Pascale Fung Kathleen McKeown Computer Science Department Columbia University, New York, NY 10027 pascale@cs.columbia.edu Abstract We propose a new algorithm, DK-vec, for aligning pairs of Asian/Indo-European noisy parallel texts without sentence boundaries. The algorithm uses frequency, position and recency information as features for pattern matching. Dynamic Time Warping is used as the matching technique between word pairs. This algorithm produces a small bilingual lexicon which provides anchor points for alignment. 1 Introduction While much work has already been done on the automatic alignment of parallel corpora (Brown et al. 1991; Kay & Röscheisen 1993; Gale & Church 1993; Church 1993; Chen 1993; Wu 1994), there are several problems which have not been fully addressed by many of these alignment algorithms. First, many corpora are noisy; segments from the source language can be totally missing from the target language or can be substituted with a target language segment which is not a translation. Similarly, the target language may include segments whose translation is totally missing from the source corpus. For example, in previous work (Church et al. 1993) on a Japanese/English corpus of the Awk manual, it was found that many picture files in the English text were totally missing from the Japanese version. In addition, programming examples were often quite different from one language to another. While such noisy corpora are basically parallel, they cannot be handled by traditional alignment programs which typically work on a sentence by sentence basis and thus cannot ignore a chunk of text in either source or target. Second, most previous alignment programs have been developed and tuned for aligning European language pairs. The particular corpora used contain sentence boundaries and these form the anchor points for aligning text (e.g., (Brown et al. 1991) and (Chen 1993) use sentence boundaries to align French and English Hansard, while (Kay & Röscheisen 1993) aligns English and German Scientific American with sentence boundaries at the sentence level). While there has been some work on aligning English with Asian languages (Wu 1994), it also relies on sentence boundaries as anchor points. Although there are tools for automatic recognition of sentence boundaries, they are not readily available and more importantly, they cannot be applied to OCR input where punctuations are often lost in noise. Furthermore, Asian languages tend to put sentence boundaries at different places than their translation in European languages. Thus, there are more many-to-one or one-to-many sentence translations. In fact, experiments with both the length-based algorithms (Wu 1994) and with char_align (Church et al. 1993) show lower performance when used on Chinese and English. This is to be expected since the syntactic structure of Asian languages is quite different from Indo-European languages and translations are also likely to be less literal than across Indo-European pairs. While Church has also noted in his work on aligning OCR input that the parallel corpora found in the real world are much less clean than the …

Content truncated to 500 words