A Statistical Model for Measuring Structural Similarity between Webpages

Proceedings of Recent Advances in Natural Language Processing, pages 24–31, Hissar, Bulgaria, Sep 7–9 2015. A Statistical Model for Measuring Structural Similarity between Webpages Zhenisbek Assylbekov, Assulan Nurkas, Inês Russinho Mouga School of Science and Technology Nazarbayev University 53 Kabanbay batyr ave., Astana, Kazakhstan {zhassylbekov, anurkas, ines.russinho}@nu.edu.kz Abstract This paper presents a statistical model for measuring structural similarity between webpages from bilingual websites. Start- ing from basic assumptions we derive the model and propose an algorithm to esti- mate its parameters in unsupervised man- ner. Statistical approach appears to bene- fit the structural similarity measure: in the task of distinguishing parallel webpages from bilingual websites our language- independent model demonstrates an F- score of 0.94–0.99 which is comparable to the results of language-dependent methods involving content similarity measures. 1 Introduction A parallel corpus is a collection of text with trans- lations into another language. Such corpora plays an important role in machine translation and multi- lingual language retrieval. Unfortunately, they are not readily available in the necessary quantities: some of them are subject to subscription or license fee and thus are not freely available, while others are domain-specific. However, there is the World Wide Web, which can be considered as one of the largest sources of parallel corpora, since there are many websites which are available in two or more languages. Many approaches have been therefore proposed for trying to exploit the Web as a parallel corpus: STRAND (Resnik and Smith, 2003), PT- Miner (Chen and Nie, 2000), BITS (Ma and Liber- man, 1999), WPDE (Zhang et al., 2006), Bitextor (Esplà-Gomis and Forcada, 2010), ILSP-FC (Pa- pavassiliou et al., 2013), etc. For most of these mining systems, there is a typical strategy for min- ing parallel texts: (1) locate bilingual websites; (2) identify parallel web pages; (3) extract bitexts. For the step (2) three main strategies can be found in the literature – they exploit: • similarities in URLs; • structural similarity of HTML files; • content-similarity of texts. Measuring structural similarity of HTML files, which is the “heart of STRAND” architecture (Resnik and Smith, 2003), involves calculating some quantitative features of candidate webpages and then comparing them to manually chosen threshold values or embedding those features into machine learning algorithms. Such approaches do not take into account the intrinsic stochastic na- ture of the mentioned features, and they require su- pervised learning of the parameters for each given website/language. In this paper we develop a more refined language-independent technique for mea- suring structural similarity between HTML pages, which uses the same amount of information as pre- vious approaches, but is more accurate in distin- guishing parallelism of webpages and can be ap- plied in unsupervised manner. 2 Related Work Measuring structural similarity between HTML files was first introduced in (Resnik, 1998), where a linearized HTML structure of candidate pairs was used to confirm parallelism of texts. Shi et al. (2006) used a file length ratio, an HTML tag similarity and a sentence alignment score to verify translational equivalence of …

Content truncated to 500 words