Exploiting Shared Chinese Characters in Chinese Word Segmentation Optimization for Chinese-Japanese Machine Translation

Exploiting Shared Chinese Characters in Chinese Word Segmentation Optimization for Chinese-Japanese Machine Translation Chenhui Chu, Toshiaki Nakazawa, Daisuke Kawahara, Sadao Kurohashi Graduate School of Informatics, Kyoto University Yoshida-honmachi, Sakyo-ku Kyoto, 606-8501, Japan {chu,nakazawa}@nlp.ist.i.kyoto-u.ac.jp {dk,kuro}@i.kyoto-u.ac.jp Abstract Unknown words and word segmentation granularity are two main problems in Chinese word segmentation for Chinese- Japanese Machine Translation (MT). In this paper, we propose an approach of exploiting common Chinese characters shared between Chinese and Japanese in Chinese word segmentation optimization for MT aiming to solve these problem- s. We augment the system dictionary of a Chinese segmenter by extracting Chinese lexicons from a parallel training corpus. In addition, we adjust the granularity of the training data for the Chinese segmenter to that of Japanese. Experimental results of Chinese-Japanese MT on a phrase-based SMT system show that our approach im- proves MT performance significantly. 1 Introduction As there are no explicit word boundary markers in Chinese, word segmentation is considered as an important first step in MT. Studies showed that a MT system with Chinese word segmentation out- performs the one treating each Chinese character as a single word, and the quality of Chinese word segmentation affects the MT performance (Xu et al., 2004; Chang et al., 2008). It has been found that besides segmentation accuracy, segmentation consistency and granularity of Chinese words are also important for MT (Chang et al., 2008). More- over, optimal Chinese word segmentation for MT is dependent on the other language, therefore, a bilingual approach is necessary (Ma and Way, 2009). c⃝ 2012 European Association for Machine Translation. 小坂 /先生/は/日本/臨床/麻酔/学会/の/ 創始/者 /である/。 小/坂 /先生/是/日本/临床/麻醉/学会/的/ 创始者 /。 Mr. Kosaka is the founder of The Japan Society for Clinical Anesthesiologists. Zh: Ja: Ref: Figure 1: Example of Chinese word segmentation problems in Chinese-Japanese MT. Most studies focus on language pairs between Chinese and other languages that have white s- paces between words (e.g. English). We focus on Chinese-Japanese MT, where segmentation is needed for both sides. Segmentation for Japanese successfully achieves F-score nearly 99% (Kudo et al., 2004), while that for Chinese is still about 95% (Wang et al., 2011). Therefore, we only do word segmentation optimization for Chinese, and keep the Japanese segmentation results. Similar to the previous works, we think the fol- lowing two problems of Chinese word segmenta- tion are important for Chinese-Japanese MT. The first problem is unknown words, which is the ma- jor difficulty faced by a Chinese segmenter affect- ing segmentation accuracy and consistency. Tak- ing “Kosaka” in Figure 1 as an example, which is a proper noun in Japanese. Because “Kosaka” is a unknown word for the Chinese segmenter, it is mistakenly segmented into two tokens, while the Japanese word segmentation result is correct. The second problem is word segmentation gran- ularity. Most Chinese segmenters adapt the famous Penn Chinese Treebank (CTB) standard (Xia et al., 2000), while most Japanese segmenters adapt a shorter unit standard. Therefore, the segmentation unit in Chinese may be longer than Japanese even for the same concept. This can …

Content truncated to 500 words