Improved Sentence Alignment on Parallel Web Pages Using a Stochastic Tree Alignment Model

Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 505–513, Honolulu, October 2008. c⃝2008 Association for Computational Linguistics Improved Sentence Alignment on Parallel Web Pages Using a Stochastic Tree Alignment Model Lei Shi Microsoft Research Asia 5F Sigma Center, 49 Zhichun Road, Beijing 100190, P. R. China leishi@microsoft.com Ming Zhou Microsoft Research Asia 5F Sigma Center, 49 Zhichun Road, Beijing 100190, P. R. China mingzhou@microsoft.com Abstract Parallel web pages are important source of training data for statistical machine translation. In this paper, we present a new approach to sentence alignment on parallel web pages. Parallel web pages tend to have parallel structures，and the structural correspondence can be indica- tive information for identifying parallel sentences. In our approach, the web page is represented as a tree, and a stochastic tree alignment model is used to exploit the structural correspondence for sentence alignment. Experiments show that this method significantly enhances alignment accuracy and robustness for parallel web pages which are much more diverse and noisy than standard parallel corpora such as “Hansard”. With improved sentence alignment performance, web mining sys- tems are able to acquire parallel sentences of higher quality from the web. 1 Introduction Sentence-aligned parallel bilingual corpora have been essential resources for statistical machine translation (Brown et al. 1993), and many other multi-lingual natural language processing applica- tions. The task of aligning parallel sentences has received considerable attention since the renais- sance of data driven machine translation in late 1980s. During the past decades, a number of methods have been proposed to address the sentence align- ment problem. Although excellent performance was reported on clean corpora, they are less robust with presence of noise. A recent study by (Singh and Husain 2005) completed a systematic evalua- tion on different sentence aligners under various conditions. Their experiments showed that the per- formance of sentence aligners are sensitive to properties of the text, such as format complexity (presence of elements other than text), structural distance (a scale from literal to free translation), the amount of noise (text deletions or preprocess- ing errors) and typological distance between lan- guages. Their performance varies on different type of texts and they all demonstrate marked perfor- mance degradation over noisy data. The results suggest that there is currently no universal solution to sentence alignment under all conditions, and different methods should be applied to different types of texts. In this paper, we specifically address sentence alignment on parallel web pages. It has come to attention with the increasing trend of acquiring large-scale parallel data from the web. Currently, large-scale parallel data are not readily available for most language pairs and domains. But due to a sharply increasing number of bilingual web sites, web mining shows great promise as a solution to this knowledge bottleneck problem. Many systems (Ma 1999; Chen 2000; Yang 2002; Resnik 2003; Chen 2004) have been developed to discover paral- lel web pages, and sentence aligners are used to extract parallel sentences from the mined web cor- pora. Sentence …

Content truncated to 500 words