High-Performance Bilingual Text Alignment Using Statistical and Dictionary Information

High-Performance Bilingual Text Alignment Using Statistical and Dictionary Information Masahiko Haruno Takefumi Yamazaki NTT Communication Science Labs. 1-2356 Take Yokosuka-Shi Kanagawa 238-03, Japan haruno@nttkb, ntt .jp yamazaki©nttkb, ntt .jp Abstract This paper describes an accurate and robust text alignment system for struc- turally different languages. Among structurally different languages such as Japanese and English, there is a limitation on the amount of word correspondences that can be statistically acquired. The proposed method makes use of two kinds of word correspondences in aligning bilin- gual texts. One is a bilingual dictionary of general use. The other is the word corre- spondences that are statistically acquired in the alignment process. Our method gradually determines sentence pairs (an- chors) that correspond to each other by re- laxing parameters. The method, by com- bining two kinds of word correspondences, achieves adequate word correspondences for complete alignment. As a result, texts of various length and of various genres in structurally different languages can be aligned with high precision. Experimen- tal results show our system outperforms conventional methods for various kinds of Japanese-English texts. 1 Introduction Corpus-based approaches based on bilingual texts are promising for various applications(i.e., lexical knowledge extraction (Kupiec, 1993; Matsumoto et al., 1993; Smadja et al., 1996; Dagan and Church, 1994; Kumano and Hirakawa, 1994; Haruno et al., 1996), machine translation (Brown and others, 1993; Sato and Nagao, 1990; Kaji et al., 1992) and infor- mation retrieval (Sato, 1992)). Most of these works assume voluminous aligned corpora. Many methods have been proposed to align bilin- gual corpora. One of the major approaches is based on the statistics of simple features such as sentence length in words (Brown and others, 1991) or in characters (Gale and Church, 1993). These tech- niques are widely used because they can be imple- mented in an efficient and simple way through dy- namic programing. However, their main targets are rigid translations that are almost literal translations. In addition, the texts being aligned were structurally similar European languages (i.e., English-French, English-German). The simple-feature based approaches don't work in flexible translations for structurally different lan- guages such as Japanese and English, mainly for the following two reasons. One is the difference in the character types of the two languages. Japanese has three types of characters (Hiragana, Katakana, and Kanji), each of which has different amounts of in- formation. In contrast, English has only one type of characters. The other is the grammatical and rhetorical difference of the two languages. First, the systems of functional (closed) words are quite differ- ent from language to language. Japanese has a quite different system of closed words, which greatly influ- ence the length of simple features. Second, due to rhetorical difference, the number of multiple match (i.e., 1-2, 1-3, 2-1 and so on) is more than that among European languages. Thus, it is impossible in gen- eral to apply the simple-feature based methods to Japanese-English translations. One alternative alignment method is the lexicon- based approach that makes use of the word- correspondence knowledge of …

Content truncated to 500 words