Complementing dictionary-based query translations with corpus statistics for cross-language IR

MT Summit VII Sept. 1999 Complementing Dictionary-Based Query Translations with Corpus Statistics for Cross-Language IR Sung Hyon Myaeng Myung-Gil Jang Dept. of Computer Science, Dept. of Knowledge Information, Chungnam National University Electronics and Telecommunications Research Institute 220 Gung-dong, Yusong-gu, 161 Kajong-dong, Yusong-gu, Taejon, Korea 305-764 Taejon, Korea 305-350 shmyaeng@cs.chungnam.ac.kr mgjang@etri.re.kr Abstract For cross-language information retrieval (CLIR), often queries or documents are translated into the other language to create a mono-lingual information retrieval situation. Having surveyed recent research results on translation-based CLIR, we have convinced ourselves that an effective query translation method is an essential element for a practical CLIR system with a reasonable quality. After summarizing the arguments and methods for query translation and survey results for dic- tionary-based translation methods, this paper describes a relatively simple yet effective method of using mutual information to handle the ambiguity problem known to be the major factor for low performance compared to mono-lingual situation. Our experimental re- sults based on the TREC-6 collection shows that this method can achieve up to 85% of the monolingual retrieval case and 96% of the manual disambiguation case. 1. Introduction Cross-language information retrieval (CLIR) en- ables a user to retrieve documents written in diverse languages using queries expressed in his or her own language. The most common approaches to CLIR have been to translate either queries or documents to over- come the language differences, although other methods without requiring translations, such as cross- language Latent Semantic Indexing (Dumais et al., 1997) have been introduced. For translation-based CLIR, query translation and document translation are two extreme approaches that have been developed while a hybrid method is possi- ble (McCarley, 1999). Query translation, a more popular method between the two, is much simpler and less expensive compared to document translation. Document translation, on the other hand, has been ar- gued to be more competitive because of rich contextu- al information in documents, compared to short queri- es whose translation may involve many errors due to unresolved ambiguities. While it is possible to apply a high-quality machine translation system for documents as in Oard & Hackett (1997), some believe that docu- ment translation is simply impractical for large-scale applications using the state-of-art technology (Car- bonell et al, 1997). A more recent development of a fast translation algorithm designed for IR makes document translation a viable option (Franz et al., 1999), but experiments have shown that a hybrid method involving both query and document translation gives the best retrieval effectiveness (McCarley, 1999). Given the assessment of the previous works in CLIR, we focus on query translation in this article because of two compelling reasons. One is that query translation is a more practical solution to CLIR. Even if we need document translation for a higher quality, query translation is still necessary for a hybrid method that has been proven to be the best so far. The other reason has to do with the difficulty of disambiguating queries, which is the main problem of query transla- tion. We believe that by …

Content truncated to 500 words