Bilingually Motivated Domain-Adapted Word Segmentation for Statistical Machine Translation

Proceedings of the 12th Conference of the European Chapter of the ACL, pages 549–557, Athens, Greece, 30 March – 3 April 2009. c⃝2009 Association for Computational Linguistics Bilingually Motivated Domain-Adapted Word Segmentation for Statistical Machine Translation Yanjun Ma Andy Way National Centre for Language Technology School of Computing Dublin City University Dublin 9, Ireland {yma, away}@computing.dcu.ie Abstract We introduce a word segmentation ap- proach to languages where word bound- aries are not orthographically marked, with application to Phrase-Based Statis- tical Machine Translation (PB-SMT). In- stead of using manually segmented mono- lingual domain-specific corpora to train segmenters, we make use of bilingual cor- pora and statistical word alignment tech- niques. First of all, our approach is adapted for the specific translation task at hand by taking the corresponding source (target) language into account. Secondly, this approach does not rely on manu- ally segmented training data so that it can be automatically adapted for differ- ent domains. We evaluate the perfor- mance of our segmentation approach on PB-SMT tasks from two domains and demonstrate that our approach scores con- sistently among the best results across dif- ferent data conditions. 1 Introduction State-of-the-art Statistical Machine Translation (SMT) requires a certain amount of bilingual cor- pora as training data in order to achieve compet- itive results. The only assumption of most cur- rent statistical models (Brown et al., 1993; Vogel et al., 1996; Deng and Byrne, 2005) is that the aligned sentences in such corpora should be seg- mented into sequences of tokens that are meant to be words. Therefore, for languages where word boundaries are not orthographically marked, tools which segment a sentence into words are required. However, this segmentation is normally performed as a preprocessing step using various word seg- menters. Moreover, most of these segmenters are usually trained on a manually segmented domain- specific corpus, which is not adapted for the spe- cific translation task at hand given that the manual segmentation is performed in a monolingual con- text. Consequently, such segmenters cannot pro- duce consistently good results when used across different domains. A substantial amount of research has been car- ried out to address the problems of word segmen- tation. However, most research focuses on com- bining various segmenters either in SMT training or decoding (Dyer et al., 2008; Zhang et al., 2008). One important yet often neglected fact is that the optimal segmentation of the source (target) lan- guage is dependent on the target (source) language itself, its domain and its genre. Segmentation con- sidered to be “good” from a monolingual point of view may be unadapted for training alignment models or PB-SMT decoding (Ma et al., 2007). The resulting segmentation will consequently in- fluence the performance of an SMT system. In this paper, we propose a bilingually moti- vated automatically domain-adapted approach for SMT. We utilise a small bilingual corpus with the relevant language segmented into basic writ- ing units (e.g. characters for Chinese or kana for Japanese). Our approach consists of using the output from an existing …

Content truncated to 500 words