An Alignment Method for Noisy Parallel Corpora based on Image Processing Techniques

An Alignment Method for Noisy Parallel Corpora based on Image Processing Techniques Jason S. Chang and Mathis H. Chen Department of Computer Science, National Tsing Hua University, Taiwan jschang@cs.nthu.edu.tw mathis @nlplab.cs.nthu.edu.tw Phone: +886-3-5731069 Fax: +886-3-5723694 Abstract This paper presents a new approach to bitext correspondence problem (BCP) of noisy bilingual corpora based on image processing (IP) techniques. By using one of several ways of estimating the lexical translation probability (LTP) between pairs of source and target words, we can turn a bitext into a discrete gray-level image. We contend that the BCP, when seen in this light, bears a striking resemblance to the line detection problem in IP. Therefore, BCPs, including sentence and word alignment, can benefit from a wealth of effective, well established IP techniques, including convolution-based filters, texture analysis and Hough transform. This paper describes a new program, PlotAlign that produces a word-level bitext map for noisy or non-literal bitext, based on these techniques. Keywords: alignment, bilingual corpus, image processing 1. Introduction Aligned corpora have proved very useful in many tasks, including statistical machine translation, bilingual lexicography (Daille, Gaussier and Lange 1993), and word sense disambiguation (Gale, Church and Yarowsky 1992; Chen, Ker, Sheng, and Chang 1997). Several methods have recently been proposed for sentence alignment of the Hansards, an English-French corpus of Canadian parliamentary debates (Brown, Lai and Mercer 1991; Gale and Church 1991a; Simard, Foster and Isabelle 1992; Chen 1993), and for other language pairs such as English-German, English-Chinese, and English-Japanese (Church, Dagan, Gale, Fung, Helfman and Satish 1993; Kay and Rtischeisen 1993; Wu 1994). The statistical approach to machine translation (SMT) can be understood as a word-by-word model consisting of two sub-models: a language model for generating a source text segment S and a translation model for mapping S to its translation T. Brown et al. (1993) also recommend using a bilingual corpus to train the parameters of Pr(S I 73, translation probability (TP) in the translation model. In the context of SMT, Brown et al. (1993) present a series of five models of Pr(S I 73 for word alignment. The authors propose using an adaptive Expectation and Maximization (EM) algorithm to estimate parameters for lexical translation probability (LTP) and distortion probability (DP), two factors in the TP, from an aligned bitext. The EM algorithm iterates between two phases to estimate LTP and DP until both functions converge. Church (1993) observes that reliably distinguishing sentence boundaries for a noisy bitext obtained from an OCR device is quite difficult. Dagan, Church and Gale (1993) recommend aligning words directly without the preprocessing phase of sentence alignment. They propose using char_align to produce a rough character-level alignment first. The rough alignment provides a basis for estimating the translation probability based on position, as well as limits the range of target words being considered for each source word. Char_align (Church 1993) is based on the observation that there are many instances of. 297 • :.-,., ---.., ~-::.~ • • :.~.".2"...'- • ...,.~. .~.- ,. • " Figure 1. Dotplot. …

Content truncated to 500 words