Text Alignment in the Real World: Improving Alignments of Noisy Translations Using Common Lexical Features, String Matching Strategies and N-Gram Comparisons

Text Alignment in the Real World: Improving Alignments of Noisy Translations Using Common Lexical Features, String Matching Strategies and N-Gram Comparisons ~ Mark W. Davis, Ted E. Dunning and William C. Ogden Computing Research Laboratory New Mexico State University Box 30001/3CRL Las Cruces, New Mexico USA { madavis,ted,ogden } @crl.nmsu.edu Abstract Alignment methods based on byte-length comparisons of alignment blocks have been remarkably successful for aligning good translations from legislative transcriptions. For noisy translations in which the parallel text of a document has significant structural differences, byte-alignment methods often do not perform well. The Pan American Health Organization (PAHO) corpus is a series of articles that were first translated by machine methods and then improved by pro- fessional translators. Many of the Spanish PAHO texts do not share formatting conven- tions with the corresponding English docu- ments, refer to tables in stylistically different ways and contain extraneous information. A method based on a dynamic programming framework, but using a decision criterion derived from a combination of byte-length ratio measures, hard matching of numbers, string comparisons and n-gram co-occur- rence matching substantially improves the performance of the alignment process. 1 Introduction Given texts in two languages that are to some degree translations of one another, an alignment of the texts associates sentences, paragraphs or phrases in one document with their translations in the other. Success- ful approaches to alignment can be divided into two primary types: those that use comparisons of lexical elements between the documents (Wu, 1994; Chen 1993; Catizone, Russell and Warwick, 1989), and IThis research was funded under DoD Contract #MDA 904-94-C-E086 those that use a statistical decision process derived from byte-length ratios between alignment blocks (Wu, 1994; Church, 1993; Gale and Church, 1991). Methods vary for the former approach, hut in the lat- ter approach, a dynamic programming framework is used to sequentially align blocks as the alignment pro- cess proceeds. Under this model, blocks are compared only with nearby blocks as the alignment proceeds, substantially reducing the computational overhead [O(n 2) ~ O(n!)] of the alignment process. In the primary literature on alignment, the texts are typically well-behaved. In byte-length ratio approaches, the presence of long stretches of blocks that have roughly similar lengths can be problematic, and some improvement can be achieved by augment- ing the byte-length measure by scores derived from lexical feature matching (Wu, 1994). When combined with radical formatting departures between docu- ments that often arise in text translations, the difficul- ties of producing good alignments are exacerbated by the presence of untranslated segments, textual rear- rangements and other problematic text features. The dynamic programming framework makes long runs of segments that have no translation in their parallel text difficult to ignore because the limited window size prevents passing over those segments to reach appro- priate areas of the document further downstream. Taken together, these difficulties can be catastrophic to the alignment process. Our experience shows that the fraction of correct alignments can drop to less than 5%. Noisy translations of this …

Content truncated to 500 words