BITS: a method for bilingual text search over the Web

MT Summit VII Sept. 1999 BITS: A Method for Bilingual Text Search over the Web Xiaoyi Ma, Mark Y. Liberman Linguistic Data Consortium 3615 Market St. Suite 200 Philadelphia, PA 19104, USA {xma,myl}@ldc.upenn.edu Abstract Parallel corpus are valuable resource for machine translation, multi-lingual text retrieval, language education and other applications, but for various reasons, its availability is very limited at present. Noticed that the World Word Web is a potential source to mine parallel text, researchers are making their efforts to explore the Web in order to get a big collection of bitext. This paper presents BITS (Bilingual Internet Text Search), a system which harvests multilingual texts over the World Wide Web with virtually no human intervention. The technique is simple, easy to port to any language pairs, and with high accuracy. The results of the experiments on German - English pair proved that the method is very successful. 1 Introduction Large parallel corpus was proved to be extremely useful for research in multilingual natural language processing and language teaching, such as statistical machine translation [Brownl990] [Melamedl998], cross-lingual information retrieval [Davis&Dunningl995] [Landauer&Littmanl990] [Oardl997], lexical acquisition [Gale&Churchl991a] [Melamedl997]. However, due to fees and copyright restrictions, for all but relatively few language pairs, parallel corpora are available only in relatively specialized forms such as United Nations proceedings [LDC], Canadian Parliament debates [LDC], and religious text and software manuals [Resnik&Melamedl997]. The available parallel corpuses are not only in relatively small size, but also unbalanced. Lack of large parallel corpus makes some research in multilingual natural language processing impossible, for example, the majority of the machine translation researches are rule-based, only a few are statistical machine translation. Some scholars believe that the lack of large parallel corpora makes statistical approach impossible, the researchers don’t have large enough parallel corpuses to give some language pairs a shot. However, the unexplored World Wide Web could be a good resource to find large-size and balanced parallel text. According to the web survey we did in 1997, 1 of 10 de domain websites are German – English bilingual, the number of de domain websites is about 150,000 at that time, so there might be 50,000 German – English bilingual websites in de domain alone. Things are changing since a potential gold mine of parallel text, the World Wide Web, has been discovered. Researchers are making their efforts to mine parallel text from the web [Resnikl998]. This paper presents a method for automatically searching parallel text on the Web. It scans a list of potential websites, finds the bilingual or multilingual websites, downloads them, cleans them up, finds pages which are translation pairs and stores them in a database. The technique is conceptually simple, easy to port to other language pairs. We evaluated the system on German – English language pair, the results indicate that the method is accurate and efficient enough to apply without human intervention. Section 2 lays out the structure of BITS. Section 3 describes the translation pair finder in detail, which is the core of …

Content truncated to 500 words