Mining Parallel Text from the Web based on Sentence Alignment

Mining Parallel Text from the Web based on Sentence Alignment * Bo Li, Juan Liu and Huili Zhu School of Computer Science, Wuhan University Wuhan, 430072, China whulibo@gmail.com, liujuan@whu.edu.cn Abstract. The parallel corpus is an important resource in the research field of data-driven natural language processing, but there are only a few parallel corpora publicly available nowadays, mostly due to the high labor force needed to construct this kind of resource. A novel strategy is brought out to automatically fetch parallel text from the web in this paper, which may help to solve the problem of the lack of parallel corpora with high quality. The system we develop first downloads the web pages from certain hosts. Then candidate parallel page pairs are prepared from the page set based on the outer features of the web pages. The candidate page pairs are evaluated in the last step in which the sentences in the candidate web page pairs are extracted and aligned first, and then the similarity of the two web pages is evaluate based on the similarities of the aligned sentences. The experiments towards a multilingual web site show the satisfactory performance of the system. Keywords: Parallel Corpus, Web Mining, Sentence Alignment, Information Extraction 1. Introduction Data-driven methods take a more important place in the research field of natural language processing nowadays, which calls an urgent need for enough parallel corpora with high quality. Parallel corpora made up with text in parallel translation are the foundational resource in data- driven natural language processing, which has a direct impact on the effectiveness of this kind of technologies such as statistical machine translation (Brown et al., 1990), cross-lingual information retrieval (Davis and Dunning, 1995; Landauer and Littman, 1990; Oard, 1997) and automatic lexical acquisition (Gale and Church, 1991b; Melamed, 1997). But it is disappointed that there are only a few parallel corpora publicly available today and most of them are small in size, specializing in narrow areas or out-of-date. It will cost too much time and human labor to construct parallel corpora with big scale and high quality, which constraints the increase of parallel corpora. The rapid development of the Internet and the intense intercourse between countries gives hope that we can construct parallel corpora automatically from the web. Through the rough observation, it is found that there are many web sites containing web pages in parallel translation. Based on the above considerations, we present a novel tool called Parallel Web Corpus Construction system (PWCC) to mine parallel corpora automatically from the web, which may help to facilitate the construction of parallel corpora with much less cost. The PWCC system uses a four-step process to fetch parallel corpora from the web. In the first step, a tool called web spider is employed to fetch all the web pages from specific hosts which * The work was finished while the first author visited Wuhan Office of Comet Electronics Hong Kong. Copyright 2007 by Bo Li, Juan Liu and Huili Zhu. 285 probably contain high-quality …

Content truncated to 500 words