Benefits of the ‘Massively Parallel Rosetta Stone’: Cross-Language Information Retrieval with over 30 Languages

Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 872–879, Prague, Czech Republic, June 2007. c⃝2007 Association for Computational Linguistics Benefits of the ‘Massively Parallel Rosetta Stone’: Cross-Language Information Retrieval with over 30 Languages Peter A. Chew Sandia National Laboratories P. O. Box 5800, MS 1012 Albuquerque, NM 87185-1012, USA pchew@sandia.gov Ahmed Abdelali New Mexico State University P.O. Box 30002, Mail Stop 3CRL Las Cruces, NM 88003-8001, USA ahmed@crl.nmsu.edu Abstract In this paper, we describe our experiences in extending a standard cross-language in- formation retrieval (CLIR) approach which uses parallel aligned corpora and Latent Semantic Indexing. Most, if not all, previous work which follows this ap- proach has focused on bilingual retrieval; two examples involve the use of French- English or English-Greek parallel cor- pora. Our extension to the approach is ‘massively parallel’ in two senses, one linguistic and the other computational. First, we make use of a parallel aligned corpus consisting of almost 50 parallel translations in over 30 distinct languages, each in over 30,000 documents. Given the size of this dataset, a ‘massively parallel’ approach was also necessitated in the more usual computational sense. Our re- sults indicate that, far from adding more noise, more linguistic parallelism is better when it comes to cross-language retrieval precision, in addition to the self-evident benefit that CLIR can be performed on more languages. 1 Introduction Approaches to cross-language information retrieval (CLIR) fall generally into one of two types, or some combination thereof: the ‘query translation’ approach or the ‘parallel corpus’ approach. The first of these, which is perhaps more common, in- volves translation of the query into the target lan- guage, for example using machine translation or on-line dictionaries. The second makes use of par- allel aligned corpora as training sets. One approach which uses parallel corpora does this in conjunc- tion with Latent Semantic Indexing (LSI) (Lan- dauer and Littman 1990, Young 1994). According to Berry et al. (1994:21), the use of LSI with paral- lel corpora can be just as effective as the query translation approach, and avoids some of the draw- backs of the latter, discussed in Nie et al. (1999). Generally, research in CLIR has not attempted to use very many languages at a time (see for ex- ample Nie and Jin 2002). With query translation (although that is not the approach that Nie and Jin take), this is perhaps understandable, as for each new language, a new translation algorithm must be included. The effort involved in extending query translation to multiple languages, therefore, is likely to be in proportion to the number of lan- guages. With parallel corpora, the reason that research has been limited to only a few languages at a time – and usually just two at a time, as in the LSI work cited above – is more likely to be rooted in the widespread perception that good parallel corpora are difficult to obtain (see for example Asker 2004). However, recent work (Resnik et al. 1999, Chew et al. …

Content truncated to 500 words