Improved Cross-Language Retrieval using Backoff Translation

Improved Cross-Language Retrieval using Backoff Translation Philip Resnik,
; Douglas Oard, ; and Gina Levow  Department of Linguistics,
Institute for Advanced Computer Studies,  College of Information Studies,  University of Maryland College Park, MD 20742 fresnik,ginag@umiacs.umd.edu, oard@glue.umd.edu ABSTRACT The limited coverageof availabletranslation lexiconscan posea se- rious challenge in some cross-language information retrieval appli- cations. We present two techniques for combining evidence from dictionary-based and corpus-based translation lexicons, and show that backoff translation outperforms a technique based on merging lexicons. 1. INTRODUCTION The effectiveness of a broad class of cross-language information retrieval (CLIR) techniques that are based on term-by-term transla- tion depends on the coverage and accuracy of the available trans- lation lexicon(s). Two types of translation lexicons are commonly used, one based on translation knowledge extracted from bilingual dictionaries [1] and the other based on translation knowledge ex- tracted from bilingual corpora [8]. Dictionaries provide reliable ev- idence, but often lack translation preference information. Corpora, by contrast, are often a better sourcefor translationsof slangor newly coined terms, but the statistical analysis through which the trans- lations are extracted sometimes produces erroneous results. In this paperwe explorethe questionof howbest to combineevidencefrom these two sources. 2. TRANSLATION LEXICONS Our term-by-term translation technique(describedbelow) requires a translation lexicon (henceforth tralex) in which each word f is as- sociated with a ranked set fe
; e  ; : : : e n g of translations. We used two translation lexicons in our experiments. 2.1 WebDict Tralex We downloadeda freely available, manuallyconstructedEnglish- French term list from the Web1 and inverted it to French-English
http://www.freedict.com . format. Since the WebDict translations appear in no particular or- der, we ranked the e i based on target language unigram statistics calculated over a large comparable corpus, the English portion of the Cross-LanguageEvaluationForum (CLEF) collection, smoothed with statistics from the Brown corpus, a balanced corpus covering many genresof English. All single-word translations are ordered by decreasing unigram frequency, followed by all multi-word transla- tions, and finally by any single-word entries not found in either cor- pus. This ordering has the effect of minimizing the effect of infre- quent words in non-standard usages or of misspellings that some- times appear in bilingual term lists. 2.2 STRAND Tralex Our secondlexical resourceis a translation lexicon obtained fully automatically via analysisof parallel French-Englishdocumentsfrom the Web. A collection of 3,378 document pairs was obtained using STRAND, our technique for mining the Web for bilingual text [7]. These document pairs were aligned internally, using their HTML markup, to produce 63,094 aligned text “chunks” ranging in length from 2 to 30 words, �8 words on average per chunk, for a total of �500K words per side. Viterbi word-alignments for these paired chunks were obtained using the GIZA implementation of the IBM statistical translation models.2 An ordered set of translation pairs was obtained by treating each alignment link between words as a co-occurrence and scoring each word pair according to the likeli- hood ratio [2]. We then rank the translation alternatives in order …

Content truncated to 500 words