Refining Word Segmentation Using a Manually Aligned Corpus for Statistical Machine Translation

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1654–1664, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Refining Word Segmentation Using a Manually Aligned Corpus for Statistical Machine Translation Xiaolin Wang Masao Utiyama Andrew Finch Eiichiro Sumita National Institute of Information and Communications Technology {xiaolin.wang,mutiyama,andrew.finch,eiichiro.sumita}@nict.go.jp Abstract Languages that have no explicit word de- limiters often have to be segmented for sta- tistical machine translation (SMT). This is commonly performed by automated seg- menters trained on manually annotated corpora. However, the word segmentation (WS) schemes of these annotated corpora are handcrafted for general usage, and may not be suitable for SMT. An analysis was performed to test this hypothesis us- ing a manually annotated word alignment (WA) corpus for Chinese-English SMT. An analysis revealed that 74.60% of the sentences in the WA corpus if segmented using an automated segmenter trained on the Penn Chinese Treebank (CTB) will contain conflicts with the gold WA an- notations. We formulated an approach based on word splitting with reference to the annotated WA to alleviate these con- flicts. Experimental results show that the refined WS reduced word alignment error rate by 6.82% and achieved the highest BLEU improvement (0.63 on average) on the Chinese-English open machine trans- lation (OpenMT) corpora compared to re- lated work. 1 Introduction Word segmentation is a prerequisite for many natural language processing (NLP) applications on those languages that have no explicit space between words, such as Arabic, Chinese and Japanese. As the first processing step, WS affects all successive steps, thus it has a large potential impact on the final performance. For SMT, the unsupervised WA, building translation models and reordering models, and decoding are all based on segmented words. Automated word segmenters built through supervised-learning methods, after decades of in- tensive research, have emerged as effective so- lutions to WS tasks and become widely used in many NLP applications. For example, the Stan- ford word segmenter (Xue et al., 2002)1 which is based on conditional random field (CRF) is em- ployed to prepare the official corpus for NTCIR- 9 Chinese-English patent translation task (Goto et al., 2011). However, one problem with applying these supervised-learning word segmenters to SMT is that the WS scheme of annotating the training cor- pus may not be optimal for SMT. (Chang et al., 2008) noticed that the words in CTB are often too long for SMT. For example, a full Chinese per- sonal name which consists of a family name and a given name is always taken as a single word, but its counterpart in English is usually two words. Manually WA corpora are precious resources for SMT research, but they used to be only avail- able in small volumes due to the production cost. For example, (Och and Ney, 2000) initially an- notated 447 English-French sentence pairs, which later became the test data set in ACL 2003 shared task on word alignment (Mihalcea and Pedersen, 2003), and was used frequently thereafter (Liang et al., 2006; …

Content truncated to 500 words