Combining Statistical Translation Techniques for Cross-Language Information Retrieval

Proceedings of COLING 2012: Technical Papers, pages 2685–2702, COLING 2012, Mumbai, December 2012. Combining Statistical Translation Techniques for Cross-Language Information Retrieval Ferhan Ture1 Jimmy Lin2,3 Douglas W. Oard2,3 (1) Department of Computer Science, University of Maryland, College Park (2) College of Information Studies, University of Maryland, College Park (3) UMIACS, University of Maryland, College Park fture@cs.umd.edu, jimmylin@umd.edu, oard@umd.edu ABSTRACT Cross-language information retrieval today is dominated by techniques that rely principally on context-independent token-to-token mappings despite the fact that state-of-the-art statistical machine translation systems now have far richer translation models available in their internal representations. This paper explores combination-of-evidence techniques using three types of statistical translation models: context-independent token translation, token translation using phrase-dependent contexts, and token translation using sentence-dependent contexts. Context-independent translation is performed using statistically-aligned tokens in parallel text, phrase-dependent translation is performed using aligned statistical phrases, and sentence-dependent translation is performed using those same aligned phrases together with an n-gram language model. Experiments on retrieval of Arabic, Chinese, and French documents using English queries show that no one technique is optimal for all queries, but that statistically significant improvements in mean average precision over strong baselines can be achieved by combining translation evidence from all three techniques. The optimal combination is, however, found to be resource-dependent, indicating a need for future work on robust tuning to the characteristics of individual collections. KEYWORDS: cross-language information retrieval, machine translation, context. 2685 1 Introduction Cross-Language Information Retrieval (CLIR) is the problem of retrieving documents relevant to a query written in a different language. There are two main approaches to tackle this problem: translating the query into the document language, or translating documents into the query language. Query translation has become the more popular approach for experimental work due to the computational feasibility of trying different system variants without repeatedly translating the entire document collection (Oard, 1998; McCarley, 1999). Query translation approaches for CLIR can be pursued either by applying a Machine Translation (MT) system or by using a token-to-token bilingual mapping, with or without translation probabilities. These approaches have complementary strengths: MT makes good use of context but at the cost of typically producing only one-best results, while token-to-token mappings can produce n-best token translations but without leveraging available contextual clues. This has led to a small cottage industry of what we might refer to as “context recovery” in which postprocessing techniques are used to select or reweight translation alternatives, usually based on evidence from term co-occurrence. We argue that this false choice between MT and n-best token-by-token translation results from thinking of MT systems as black boxes. A modern statistical MT system has internally a series of increasingly rich representations that are exploited during the training and decoding processes. First, token alignments are generated for each training sentence pair, a process that also creates context-independent token-to-token translation probabilities. Second, these alignments are generalized to learn a Synchronous Context-Free Grammar (SCFG), in which probabilistic rules describe the translation of larger units of text. Finally, the translation grammar is combined with a …

Content truncated to 500 words