Can Word Segmentation be Considered Harmful for Statistical Machine Translation Tasks between Japanese and Chinese?

Copyright 2012 by Jing Sun and Yves Lepage 26th Pacific Asia Conference on Language,Information and Computation pages 351–360 Can Word Segmentation be Considered Harmful for Statistical Machine Translation Tasks between Japanese and Chinese? Jing Sun and Yves Lepage NLP Laboratory / Hibikino 2-7, Wakamatsu-ku Graduate School of IPS / Kitakyushu-shi, Fukuoka-ken Waseda University / Japan 808-0135 {cecily.sun@akane., yves.lepage@}waseda.jp Abstract Unlike most Western languages, there are no typographic boundaries between words in written Japanese and Chinese. Word seg- mentation is thus normally adopted as an ini- tial step in most natural language processing tasks for these Asian languages. Although word segmentation techniques have improved greatly both theoretically and practically, there still remains some problems to be tackled. In this paper, we present an effective approach in extracting Chinese and Japanese phrases without conducting word segmentation be- forehand, using a sampling-based multilingual alignment method. According to our exper- iments, it is also feasible to train a statis- tical machine translation system on a small Japanese-Chinese training corpus without per- forming word segmentation beforehand. 1 Introduction Unlike most European languages, there are no ex- plicit typographic boundaries like white spaces be- tween words in many written Asian languages such as Chinese, Japanese, Korean, Thai, Lao and Viet- namese. Therefore, word segmentation for such languages is usually the first important step in most Natural Language Processing (NLP) applica- tions especially in statistical machine translation. Although word segmentation techniques have im- proved greatly in recent years, there are still some difficulties that remain to be addressed. Word segmentation schemes are not system- independent, application-independent nor language- independent. Different Chinese Word Segmentation (CWS) tools applied to the same Chinese sentence may lead to different results depending on their seg- mentation. For instance, 学生会 (pinyin: xu´e sh¯eng hu`ı) in Chinese may be interpreted as 学生 会 ‘stu- dent(s) can (do)’ or 学生会 ‘Students’ Union’ re- spectively. Figure 1 gives an example of pre-segmented text and unsegmented text in both Chinese and Japanese. We applied four CWS tools: Urheen (Wang et al., 2010), ICTCLAS (Zhang et al., 2003) and Stand- ford Chinese word segmenter (Tseng et al., 2005) trained on CTB and PKU. This example clearly shows that word segmentation tools may do harm to cross-lingual tasks, because: (i) there may be inconsistencies of segmentation results across languages such as different sizes of granularity in Japanese and Chinese; (ii) for the same language, different word segmen- tation tools may produce different results; (iii) the same word segmentation tool trained on different corpora may produce different re- sults. Such inconsistencies lead to increased error rates in Statistical Machine Translation. Significant improvements in Chinese word seg- mentation techniques have been obtained recently and reported accuracy rates (compared to those of human Golden Standard) have reached 98%. How- ever, for cross-lingual NLP tasks, such as phrasal ex- traction or Machine Translation, Zhang et al. (2008) showed that even the most accurate word segmen- tation may not produce the best translation out- 351 Original Chinese sentence: 没事先约好，白跑了回津屋崎。 Translation in Japanese: 事前予約をしなかったので、むだに津屋崎に行きました。 Meaning …

Content truncated to 500 words