Feature-Based Method for Document Alignment in Comparable News Corpora

Proceedings of the 12th Conference of the European Chapter of the ACL, pages 843–851, Athens, Greece, 30 March – 3 April 2009. c⃝2009 Association for Computational Linguistics Feature-based Method for Document Alignment in Comparable News Corpora Thuy Vu, Ai Ti Aw, Min Zhang Department of Human Language Technology, Institute for Infocomm Research 1 Fusionopolis Way, #21-01 Connexis, South Tower, Singapore 138632 {tvu, aaiti, mzhang}@i2r.a-star.edu.sg Abstract In this paper, we present a feature-based me- thod to align documents with similar content across two sets of bilingual comparable cor- pora from daily news texts. We evaluate the contribution of each individual feature and investigate the incorporation of these diverse statistical and heuristic features for the task of bilingual document alignment. Experimental results on the English-Chinese and English- Malay comparable news corpora show that our proposed Discrete Fourier Transform- based term frequency distribution feature is very effective. It contributes 4.1% and 8% to performance improvement over Pearson’s correlation method on the two comparable corpora. In addition, when more heuristic and statistical features as well as a bilingual dic- tionary are utilized, our method shows an ab- solute performance improvement of 23.2% and 15.3% on the two sets of bilingual corpo- ra when comparing with a prior information retrieval-based method. 1 Introduction The problem of document alignment is described as the task of aligning documents, news articles for instance, across two corpora based on content similarity. The groups of corpora can be in the same or in different languages, depending on the purpose of one’s task. In our study, we attempt to align similar documents across comparable cor- pora which are bilingual, each set written in a different language but having similar content and domain coverage for different communication needs. Previous works on monolingual document alignment focus on automatic alignment between documents and their presentation slides or be- tween documents and their abstracts. Kan (2007) uses two similarity measures, Cosine and Jac- card, to calculate the candidate alignment score in his SlideSeer system, a digital library software that retrieves documents and their narrated slide presentations. Daumé and Marcu (2004) use a phrase-based HMM model to mine the alignment between documents and their human-written ab- stracts. The main purpose of this work is to in- crease the size of the training corpus for a statistical-based summarization system. The research on similarity calculation for mul- tilingual comparable corpora has attracted more attention than monolingual comparable corpora. However, the purpose and scenario of these works are rather varied. Steinberger et al. (2002) represent document contents using descriptor terms of a multilingual thesaurus EUROVOC1, and calculate the semantic similarity based on the distance between the two documents’ representa- tions. The assignment of descriptors is trained by log-likelihood test and computed by �����, Co- sine, and Okapi. Similarly, Pouliquen et al. (2004) use a linear combination of three types of knowledge: cognates, geographical place names reference, and map documents based on the EUROVOC. The major limitation of these works is the use of EUROVOC, which is a specific re- …

Content truncated to 500 words