An Iterative Link-based Method for Parallel Web Page Mining

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1216–1224, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics An Iterative Link-based Method for Parallel Web Page Mining Le Liu1, Yu Hong1, Jun Lu2, Jun Lang2, Heng Ji3, Jianmin Yao1 1School of Computer Science & Technology, Soochow University, Suzhou, 215006, China 2Institute for Infocomm Research, Singapore, 138632 3Computer Science Department, Rensselaer Polytechnic Institute, Troy, NY 12180, USA giden@sina.cn,{tianxianer,lujun59,billlangjun}@gmail.com jih@rpi.edu,jyao@suda.edu.cn Abstracts Identifying parallel web pages from bi- lingual web sites is a crucial step of bi- lingual resource construction for cross- lingual information processing. In this paper, we propose a link-based approach to distinguish parallel web pages from bi- lingual web sites. Compared with the ex- isting methods, which only employ the internal translation similarity (such as content-based similarity and page struc- tural similarity), we hypothesize that the external translation similarity is an effec- tive feature to identify parallel web pages. Within a bilingual web site, web pages are interconnected by hyperlinks. The basic idea of our method is that the trans- lation similarity of two pages can be in- ferred from their neighbor pages, which can be adopted as an important source of external similarity. Thus, the translation similarity of page pairs will influence each other. An iterative algorithm is de- veloped to estimate the external transla- tion similarity and the final translation similarity. Both internal and external similarity measures are combined in the iterative algorithm. Experiments on six bilingual websites demonstrate that our method is effective and obtains signifi- cant improvement (6.2% F-Score) over the baseline which only utilizes internal translation similarity. 1 Introduction Parallel corpora have played an important role in multilingual Natural Language Processing, espe- cially in Machine Translation (MT) and Cross- lingual Information Retrieval(CLIR). However, it’s time-consuming to build parallel corpora manually. Some existing parallel corpora are subject to subscription or license fee and thus not freely available, while others are domain-specific. Therefore, a lot of previous research has focused on automatically mining parallel corpora from the web. In the past decade, there have been extensive studies on parallel resource extraction from the web (e.g., Chen and Nie, 2000; Resnik 2003; Jiang et al., 2009) and many effective Web min- ing systems have been developed such as STRAND, PTMiner, BITS and WPDE. For most of these mining systems, there is a typical paral- lel resource mining strategy which involves three steps: (1) locate the bilingual websites (2) identi- fy parallel web pages from these bilingual web- sites and (3) extract bilingual resources from the parallel web pages. In this paper, we focus on the step (2) which is regarded as the core of the mining system (Chunyu, 2007). Estimating the translation simi- larity of two pages is the most basic and key problem in this step. Previous approaches have tried to tackle this problem by using the infor- mation within the pages. For example, in the STRAND and PTMiner system, a structural fil- tering process that relies on the analysis …

Content truncated to 500 words