Using Mutual Information to Resolve Query Translation Ambiguities and Query Term Weighting

Using Mutual Information to Resolve Query Translation Ambiguities and Query Term Weighting 1 Myung-Gil Jang, 2 Sung Hyon Myaeng and 1 Se Young Park 1 Dept. of Knowledge Information, Electronics and Telecommunications Research Institute 161 Kajong-Dong, Yusong-Gu, Taejon, Korea 305-350 { mgjang, sypark } @etri.re.kr 2 Dept. of Computer Science, Chungnam National University 220 Gung-Dong, Yusong-Gu, Taejon, Korea 305-764 shmyaeng@cs.chungnam.ac.kr Abstract An easy way of translating queries in one language to the other for cross-language information retrieval (IR) is to use a simple bilingual dictionary. Because of the general- purpose nature of such dictionaries, however, this simple method yields a severe translation ambiguity problem. This paper describes the degree to which this problem arises in Korean-English cross-language IR and suggests a relatively simple yet effective method for disambiguation using mutual information statistics obtained only from the target document collection. In this method, mutual information is used not only to select the best candidate but also to assign a weight to query terms in the target language. Our experimental results based on the TREC-6 collection shows that this method can achieve up to 85% of the monolingual retrieval case and 96% of the manual disambiguation case. Introduction Cross-language information retrieval (IR) enables a user to retrieve documents written in diverse languages using queries expressed in his or her own language. For cross-language IR, either queries or documents are translated to overcome the language differences. Although it is possible to apply a high-quality machine translation system for documents as in Oard & Hackett (1997), query translation has emerged as a more popular method because it is much simpler and more economical compared to document translation. Query translation can be done in one or more of the three approaches: a dictionary-based approach, a thesaurus-based approach, or a corpus-based approach. There are three problems that a cross-language IR system using a query translation method must solve (Grefenstette, 1998). The first problem is to figure out how a term expressed in one language might be written in another. The second problem is to determine which of the possible translations should be retained. The third problem is to determine how to properly weight the importance of translation alternatives when more than one is retained. For cross-language IR between Korean and English, i.e. between Korean queries and English documents, an easy way to handle query , translation is to use a Korean-English machine- readable dictionary (MRD) because such bilingual MRDs are more widely available than other resources such as parallel corpora. However, it has been known that with a simple use of bilingual dictionaries in other language pairs, retrieval effectiveness can be only 40%- 60% of that with monolingual retrieval (Ballesteros & Croft, 1997). It is obvious that other additional resources need to be used for better performance. This paper focuses on the last two problems: pruning translations and calculating the weights for translation alternatives. We first describe the overall query translation process and the extent to which the ambiguity problem arises in Korean-English cross-language IR. We …

Content truncated to 500 words