Automatic construction of parallel English-Chinese corpus for cross-language information retrieval

Automatic construction of parallel English-Chinese corpus for cross-language information retrieval Jiang Chen and Jian-Yun Nie D~partement d'Informatique et Recherche Op~rationnelle Universit~ de Montreal C.P. 6128, succursale CENTRE-VILLE Montreal (Quebec), Canada H3C 3J7 {chen, nie} @iro. umontreal, ca Abstract A major obstacle to the construction of a probabilis- tic translation model is the lack of large parallel cor- pora. In this paper we first describe a parallel text mining system that finds parallel texts automatically on the Web. The generated Chinese-English paral- lel corpus is used to train a probabilistic translation model which translates queries for Chinese-English cross-language information retrieval (CLIR). We will discuss some problems in translation model training and show the preliminary CUR results. 1 Introduction Parallel texts have been used in a number of studies in computational linguistics. Brown et al. (1993) defined a series of probabilistic translation models for MT purposes. While people may question the effectiveness of using these models for a full-blown MT system, the models are certainly valuable for de- veloping translation assistance tools. For example, we can use such a translation model to help com- plete target text being drafted by a human transla- tor (Langlais et al., 2000). Another utilization is in cross-language informa- tion retrieval (CLIR) where queries have to be trans- lated from one language to another language in which the documents are written. In CLIR, the qual- ity requirement for translation is relatively low. For example, the syntactic aspect is irrelevant. Even if the translated word is not a true translation but is strongly related to the original query, it is still help- ful. Therefore, CLIR is a suitable application for such a translation model. However, a major obstacle to this approach is the lack of parallel corpora for model training. Only a few such corpora exist, including the Hansard English-French corpus and the HKUST English- Chinese corpus (Wu, 1994). In this paper, we will describe a method which automatically searches for parallel texts on the Web. We will discuss the text mining algorithm we adopted, some issues in trans- lation model training using the generated parallel corpus, and finally the translation model's perfor- mance in CLIR. 2 Parallel Text Mining Algorithm The PTMiner system is an intelligent Web agent that is designed to search for large amounts of paral- lel text on the Web. The mining algorithm is largely language independent. It can thus be adapted to other language pairs with only minor modifications. Taking advantage of Web search engines as much as possible, PTMiner implements the following steps (illustrated in Fig. 1): 1 Search for candidate sites - Using existing Web search engines, search for the candidate sites that may contain parallel pages; 2 File name fetching - For each candidate site, fetch the URLs of Web pages that are indexed by the search engines; 3 Host crawling - Starting from the URLs col- lected in the previous step, search through each candidate site separately for more URLs; 4 Pair scan - From the obtained URLs of …

Content truncated to 500 words