Aligning Parallel Bilingual Corpora Statistically with Punctuation Criteria

Computational Linguistics and Chinese Language Processing Vol. 10, No. 1, March 2005, pp. 95-122 95 © The Association for Computational Linguistics and Chinese Language Processing Aligning Parallel Bilingual Corpora Statistically with Punctuation Criteria Thomas C. Chuang∗ and Kevin C. Yeh+ Abstract We present a new approach to aligning sentences in bilingual parallel corpora based on punctuation, especially for English and Chinese. Although the length-based approach produces high accuracy rates of sentence alignment for clean parallel corpora written in two Western languages, such as French-English or German-English, it does not work as well for parallel corpora that are noisy or written in two disparate languages such as Chinese-English. It is possible to use cognates on top of the length-based approach to increase the alignment accuracy. However, cognates do not exist between two disparate languages, which limit the applicability of the cognate-based approach. In this paper, we examine the feasibility of exploiting the statistically ordered matching of punctuation marks in two languages to achieve high accuracy sentence alignment. We have experimented with an implementation of the proposed method on parallel corpora, the Chinese-English Sinorama Magazine Corpus and Scientific American Magazine articles, with satisfactory results. Compared with the length-based method, the proposed method exhibits better precision rates based on our experimental reuslts. Highly promising improvement was observed when both the punctuation-based and length-based methods were adopted within a common statistical framework. We also demonstrate that the method can be applied to other language pairs, such as English-Japanese, with minimal additional effort. Keywords: Sentence Alignment, Cognate Alignment, Machine Translation ∗ Department of Computer Science, Vanung University, No. 1 Van-Nung Road, Chung-Li Tao-Yuan, Taiwan, ROC E-mail: tomchuang@cc.vit.edu.tw + Department of Computer Science, National Tsing Hua University, 101, Kuangfu Road, Hsinchu, 300, Taiwan, ROC 96 Thomas C. Chuang and Kevin C. Yeh 1. Introduction Bilingual corpora are very important for building natural language processing systems [Moore 2002; Gey et al. 2002], including data-driven machine translation [Dolan et al. 2002], computer-assisted revision of translations [Jutras 2000], and cross-language information retrieval [Chen and Gey 2001]. In order to develop NLP systems, it is useful to align bilingual corpora at the sentence level with very high precision [Moore 2002; Chuang et al. 2002, Kueng and Su 2002]. With aligned sentences, further analysis such as phrase and word alignment analysis [Ker and Chang 1997; Melamed 1997], bilingual terminology [Déjean et al. 2002] and collocation [Wu 200] extraction analysis can be performed. Yang, C, Li, K. [2003] proposed an alignment method for bilingual title pairs on the Web for automatic generation of bilingual parallel corpora. The hybrid dictionary approach [Collier et al. 1998], text-based alignment [Kay and Röscheisen 1993], part of speech-based alignment [Chen and Chen 1994], and the lexical method [Chen 1993] are other examples of sentence alignment methods. While these methods presume little or no prior knowledge of source and target languages, they are relatively complex and require significant amounts of parallel text and language resources. Much work reported in the computational linguistics literature has focused on aligning English-French …

Content truncated to 500 words