Rapidly Retargetable Interactive Translingual Retrieval

Rapidly Retargetable Interactive Translingual Retrieval Gina-Anne Levow Institute for Advanced Computer Studies University of Maryland, College Park, MD 20742 gina@umiacs.umd.edu Douglas W. Oard College of Information Studies Institute for Advanced Computer Studies University of Maryland, College Park, MD 20742 oard@glue.umd.edu Philip Resnik Department of Linguistics Institute for Advanced Computer Studies University of Maryland, College Park, MD 20742 resnik@umiacs.umd.edu ABSTRACT This paper describes a system for rapidly retargetable interactive translingual retrieval. Basic functionality can be achieved for a new document language in a single day, and further improvements re- quire only a relatively modest additional investment. We applied the techniquesfirst to searchChinesecollectionsusingEnglish queries, and have successfullyaddedFrench, German, and Italian document collections. We achievethis capability throughseparationof language- dependent and language-independent components and through the application of asymmetric techniquesthat leveragean extensiveEn- glish retrieval infrastructure. Keywords Cross-language information retrieval 1. INTRODUCTION Our goalis to producesystemsthat allow interactiveusersto present English queries and retrieve documents in languages that they can- not read. In this paper we focus on what we call “rapid retargetabil- ity”: extending interactive translingual retrieval functionality for a new documentlanguagerapidly with few language-specificresources. Our current system can be retargeted to a new language in one day with only one language-dependent resource: a bilingual term list.1 Our language-independent architecture consists of two main com- ponents: 1. Document translation and indexing 2. Interactive retrieval We describe each of these components, demonstrate their effective- nessfor information retrieval tasks, andthen concludeby describing our experience with adding French, German and Italian document collections to a system that was originally developed for Chinese.
For Asian languageswe also use a language-specificsegmentation system. . 2. DOCUMENT TRANSLATION AND IN- DEXING We haveadopted a documenttranslation architecture for two rea- sons. First, we support a single query language(English) but multi- ple documentlanguages,so indexingEnglish terms simplifies query processing(where interactive responsetime can be a concern). Sec- ond, a document translation architecture simplifies the display of translated documentsby decoupling the translation and display pro- cesses. Gigabyte collections require machine translation that is or- ders of magnitude faster than present commercial systems. We ac- complishthis usingterm-by-term translation, in which the basicdata structure is a simple hash table lookup. Any translation requires some source of translation knowledge—we use a bilingual term list containingEnglishtranslation(s) for eachforeign languageterm. We typically construct these term lists by harvesting Internet-available translation resources, so the foreign languageterms for which trans- lations are known are typically an eclectic mix of root and inflected forms. We accommodate this limitation using a four-stage backoff statistical stemming approach to enhance translation coverage. 2.1 Preprocessing. Differences in use of diacritic-s, case, and punctuationcan inhibit matching between term list entries and document terms, so normal- ization is important. In order to maximize the probability of match- ing document words with term list entries, we normalize the bilin- gual term list and the documents by: � converting characters in Western languages to lowercase, � removing all accents and diacritics, and � segmentation, which for Western languages merely involves separatingpunctuationfrom other text by the additionof white space. Our preprocessingalsoincludesconversionof the bilingual term …

Content truncated to 500 words