Web-Based Query Translation for English-Chinese CLIR

Computational Linguistics and Chinese Language Processing Vol. 13, No. 1, March 2008, pp. 61-90 61 © The Association for Computational Linguistics and Chinese Language Processing [Received June 30, 2007; Revised October 8, 2007; Accepted January 12, 2008] Web-Based Query Translation for English-Chinese CLIR Chengye Lu∗, Yue Xu∗, and Shlomo Geva∗ Abstract Dictionary-based translation is a traditional approach in use by cross-language information retrieval systems. However, significant performance degradation is often observed when queries contain words that do not appear in the dictionary. This is called the Out of Vocabulary (OOV) problem. In recent years, Web mining has been shown to be one of the effective approaches for solving this problem. However, the questions of how to extract Multiword Lexical Units (MLUs) from the Web content and how to select the correct translations from the extracted candidate MLUs are still two difficult problems in Web mining based automated translation approaches. Most statistical approaches to MLU extraction rely on statistical information extracted from huge corpora. In the case of using Web mining techniques for automated translations, these approaches do not perform well because the size of the corpus is usually too small and statistical approaches that rely on a large sample can become unreliable. In this paper, we present a new Chinese term measurement and a new Chinese MLU extraction process that work well on small corpora. We also present our approach to the selection of MLUs in a more accurate manner. Our experiments show marked improvement in translation accuracy over other commonly used approaches. Keywords: Cross-Language Information Retrieval, CLIR, Query Translation, Web Mining, OOV Problem, Term Extraction 1. INTRODUCTION As more and more documents written in various languages become available on the Internet, users increasingly wish to explore documents that were written in either their native language ∗ Faculty of Information Technology, School of Software Engineering and Data Communications, Queensland University of Technology, Brisbane, QLD 4001, Australia E-mail: {c.lu,yue.xu,s.geva}@qut.edu.au 62 Chengye Lu et al. or some language other than English. Cross-language information retrieval (CLIR) systems allow users to retrieve documents written in more than one language through queries written in a different language. This is a helpful end-user feature. Obviously, translation is needed in the CLIR process; either translating the query into the document language, or translating the documents into the query language. The common approach is to translate the query into the document language using a dictionary. Dictionary-based translation has been adopted in cross-language information retrieval because bilingual dictionaries are widely available, dictionary-based approaches are easy to implement, and the efficiency of word translation with a dictionary is high. However, due to the vocabulary limitation of dictionaries, very often the translations of some words in a query cannot be found in a dictionary. This problem is called the Out of Vocabulary (OOV) problem. Very often, the OOV terms are proper names or newly created words. Even using the best dictionary, the OOV problem is unavoidable. As input queries are usually short, query expansion does not provide enough information to help …

Content truncated to 500 words