Learning an English-Chinese Lexicon from a Parallel Corpus

LEARNING AN ENGLISH-CHINESE LEXICON FROM A PARALLEL CORPUS Dekai Wu Xuanyin Xia HKUST Department of Computer Science University of Science & Technology Clear Water Bay, Hong Kong {dekai,samxia}@cs.ust.hk Abstract We report experiments on automatic learning of an English-Chinese translation lexicon, through statistical training on a large parallel corpus. The learned vocabulary size is non- trivial at 6,517 English words averaging 2.33 Chinese translations per entry, with a manually- filtered precision of 95.1% and a single-most-probable precision of 91.2%. We then introduce a significance filtering method that is fully automatic, yet still yields a weighted precision of 86.0%. Learning of translations is adaptive to the domain. To our knowledge, these are the first empirical results of the kind between an Indo-European and non-Indo-European language for any significant corpus size with a non-toy vocabulary. 1 Introduction A criticism of statistical machine translation tools is that convincing empirical results to date are largely confined to similar language pairs, such as French and English. We offer some contribu- tions to the pool of evidence supporting the language-independence of statistical techniques, from the SILC project at HKUST which is studying machine learning of natural language translation. Specifically, we report accuracy rates for a new performance measure: the precision of statistical translation lexicon acquisition, between English and Chinese. In the first phase of SILC (statistical inter-lingual conversion), we have (1) collected a bilingual corpus of parallel English and Chinese text, (2) aligned the sentences within the corpus, and (3) learned a bilingual lexicon from the aligned data, to be embedded within learned translation models. Preliminary results on the first two steps were reported in Wu (1994); an updated summary is given in section 2, before we discuss the bilingual lexicon learning. One motivation for this work is to conduct English-Chinese "acid tests" of statistical NLP techniques. Another benefit of the approach is that it obtains not only a translation lexicon, but also the probabilities of alternative translations for the same word. A further advantage of the learning approach is the ability to acquire lexicons that are adapted for particular domains or genres. The vocabulary of our corpus, for example, includes a high proportion of words not found in the English-Chinese machine-readable dictionaries we have seen. 2 A Sentence-Aligned Corpus Though large parallel bilingual corpora are relatively scarce compared with monolingual corpora, they have generated more interesting results. Significant progress has been made on problems 206 including automatic sentence alignment (Kay & Röscheisen 1988; Catizone et al. 1989; Gale & Church 1991; Brown et al. 1991; Chen 1993), coarse alignment (Church 1993; Fung & Church 1994), statistical machine translation (Brown et al. 1990; Brown et al. 1993), word alignment (Dagan et al. 1993), word sense disambiguation (Gale et al. 1993), and collocation learning (Smadja & McKeown 1994), all exploiting parallel corpora. To facilitate empirical studies that cannot rely on shared characteristics of Indo-European languages, we have been constructing the HKUST English-Chinese Parallel Bilingual Corpus. Currently, the material in the corpus consists primarily of fairly tight, literal sentence …

Content truncated to 500 words