Web based English-Chinese OOV term translation using Adaptive rules and Recursive feature selection

Web based English-Chinese OOV term translation using Adaptive rules and Recursive feature selection Jian Qu, Nguyen Le Minh, Akira Shimazu School of Information Science, JAIST Ishikawa, Japan 923-1292 qujian@jaist.ac.jp Abstract. Cross-Language Information Retrieval (CLIR) system uses dictionaries for information retrieval. However, out of vocabulary (OOV) terms cannot be found in dictionaries. Although many researchers in the past have endeavored to solve the OOV term translation problem, but little attention has been paid to hybrid translations “α1- antitrypsin deficiency (α1-抗胰蛋白酶缺乏症)”. This paper presents a novel OOV term translation mining approach, which proposes a new adaptive rules system for hybrid translations and a new recursive feature selection method for supervised machine learning. We evaluate the propose method with English-Chinese OOV term translation. Our experiments show that adaptive rules system and recursive feature selection with Bayesian Net can significantly outperform existing supervised models. Keywords: adaptive rules, recursive feature selection. 1 Introduction CLIR system employs multilingual dictionaries for translating queries from one to another. To ensure a perfect CLIR system, it must use a perfect multilingual dictionary. However, a perfect multilingual dictionary would never be existed due to OOV terms. OOV terms are typically new terms that cannot be found in dictionaries, such as personal names, place names, new technical terms and translated words etc. New OOV terms are emerging every day, this makes it difficult for multilingual dictionaries to cover all translations of OOV terms. Generally speaking, OOV terms can be translated by three methods, they are: human translation with field of specified knowledge; automatic translation from parallel corpus or comparable corpus (Zhou, Truran et al. 2008); and automatic mining from the Internet (Lu, Chien et al. 2004). Most OOV terms have their correspondent human translation nearby on the Internet (Cheng, Teng et al. 2004; Zhang, Huang et al. 2005; Udupa, K et al. 2009). The quality of the translation is incomparable between human translation and automatic translation. However, automatic translation offers the speed and low cost over human translation. While automatic mining takes the advantage of both human and automatic translation. The translations mined from the Internet are usually high quality and require low man power cost. According to our observation, some translations of the technical OOV term are hybrid translations. Hybrid translations use part target language and part source language, for example, a English OOV term: “DiGeorge's syndrome” and its Chinese translation “DiGeorge's 症候群”. “DiGeorge's” are source languages and “症候群” are target languages. Solving Hybrid translations is important because existing approaches would retrieve “ 症 候 群 ” as the translation of “DiGeorge's syndrome”. If this translation is applied to CLIR, many disease documents unrelated to “DiGeorge's syndrome” will be retrieved, because many Chinese medical terms end with the term “症候群”. In this paper, we propose an OOV term translation method that employs the automatic mining, a novel adaptive rules system for hybrid translations, and a novel recursive feature selection using supervised machine learning by Bayesian net with Adaboost. 25th Pacific Asia Conference on Language, Information and Computation, pages 1–10 1 …

Content truncated to 500 words