Compiling Bilingual Lexicon Entries From a Non-Parallel English-Chinese Corpus

i Compiling Bilingual Lexicon Entries From a Non-Parallel English-Chinese Corpus Pascale Fung 1 Computer Science Department Columbia University New York, NY 10027 pascale@cs, columbia, edu Abstract We propose a novel context heterogeneity similarity measure between words and their translations in helping to compile bilingual lexicon entries from a non-parallel English-Chinese corpus. Current algorithms for bilingual lexicon compilation rely on occurrence frequencies, length or positional statistics derived from parallel texts. There is little correlation between such statistics of a word and its translation in non-parallel corpora. On the other hand, we suggest that words with productive context in one language translate to words with productive context in another language, and words with rigid context translate into words With rigid context. Context heterogeneity measures how productive the context of a word is in a given domain, independent of its absolute occurrence frequency in the text. Based on this information, we derive statistics of bilingual word pairs from a non-parallel corpus. These statistics can be used to bootstrap a bilingual dictionary compilation algorithm. 1 Introduction Building a domain-specific bilingual lexicon is one significant component in machine translation and machine-aided translation systems. These terms are often not found in standard dictionaries. Human translators, not being experts in every technical or regional domain, cannot produce their translations effec- tively. Automatic:compilation of such a bilingual lexicon in specific domains is therefore highly desirable. We present an algorithm in finding word correlation statistics for automatic bilingual lexicon compilation from a non-parallel corpus in Chinese and English. Most previous automatic lexicon compilation techniques require a sentence-aligned clean parallel bilingual corpus (Kupiec 1993; Smadja & McKeown 1994; Kumano & Hirakawa 1994; Dagan et al. 1993; Wu & Xia 1994). We have previously shown an algorithm which extracts a bilingual lexicon from noisy parallel corpus without sentence alignment (Fung & McKeown 1994; Fung 1995). Although bilingual parallel corpora have been available in recent years, they are still relatively few in comparison to the large amount of monolingual text. Acquiring and processing of parallel corpora are usually labour-intensive and time-consuming. More importantly, the existence of a parallel corpus in a particular domain means some translator has translated it, therefore, the bilingual lexicon compiled from such a corpus is at best a reverse engineering of the lexicon this translator used. On the other hand, if we can compile a dictionary of domain-specific words from non-parallel corpora of monolingual texts, the results would be much more meaningful and useful. This research is partly supported by the Department of Electrical and Electronic Engineering, Hong Kong University of Science and Technology. 173 As demonstrated in all the bilingual lexicon compilation algorithms, the foremost task is to identify word features which are similar between a word and its translation, yet different between a word and other words which are not its translations. In parallel corpora, this feature could be the positional co-occurrence of a word and its translation in the other language in the same sentences (Kupiec 1993; Smadja & McKeown 1994; Kumano & Hirakawa 1994; …

Content truncated to 500 words