Mining the Web for Bilingual Text

Mining the Web for Bilingual Text Philip Resnik* Dept. of Linguistics/Institute for Advanced Computer Studies University of Maryland, College Park, MD 20742 resnik@umiacs, umd. edu Abstract STRAND (Resnik, 1998) is a language- independent system for automatic discovery of text in parallel translation on the World Wide Web. This paper extends the prelim- inary STRAND results by adding automatic language identification, scaling up by orders of magnitude, and formally evaluating perfor- mance. The most recent end-product is an au- tomatically acquired parallel corpus comprising 2491 English-French document pairs, approxi- mately 1.5 million words per language. 1 Introduction Text in parallel translation is a valuable re- source in natural language processing. Sta- tistical methods in machine translation (e.g. (Brown et al., 1990)) typically rely on large quantities of bilingual text aligned at the doc- ument or sentence level, and a number of approaches in the burgeoning field of cross- language information retrieval exploit parallel corpora either in place of or in addition to map- pings between languages based on information from bilingual dictionaries (Davis and Dunning, 1995; Landauer and Littman, 1990; Hull and Oard, 1997; Oard, 1997). Despite the utility of such data, however, sources of bilingual text are subject to such limitations as licensing restric- tions, usage fees, restricted domains or genres, and dated text (such as 1980's Canadian poli- tics); or such sources simply may not exist for * This work was supported by Department of De- fense contract MDA90496C1250, DARPA/ITO Con- tract N66001-97-C-8540, and a research grant from Sun Microsystems Laboratories. The author gratefully ac- knowledges the comments of the anonymous reviewers, helpful discussions with Dan Melamed and Doug Oard, and the assistance of Jeff Allen in the French-English experimental evaluation. language pairs of interest. Although the majority of Web content is in English, it also shows great promise as a source of multilingual content. Using figures from the Babel survey of multilinguality on the Web (htZp ://www. isoc. org/), it is possible to esti- mate that as of June, 1997, there were on the or- der of 63000 primarily non-English Web servers, ranging over 14 languages. Moreover, a follow- up investigation of the non-English servers sug- gests that nearly a third contain some useful cross-language data, such as parallel English on the page or links to parallel English pages -- the follow-up also found pages in five languages not identified by the Babel study (Catalan, Chi- nese, Hungarian, Icelandic, and Arabic; Michael Littman, personal communication). Given the continued explosive increase in the size of the Web, the trend toward business organizations that cross national boundaries, and high levels of competition for consumers in a global mar- ketplace, it seems impossible not to view mul- tilingual content on the Web as an expanding resource. Moreover, it is a dynamic resource, changing in content as the world changes. For example, Diekema et al., in a presentation at the 1998 TREC-7 conference (Voorhees and Har- man, 1998), observed that the performance of their cross-language information retrieval was hurt by lexical gaps …

Content truncated to 500 words