Consistent Word Segmentation, Part-of-Speech Tagging and Dependency Labelling Annotation for Chinese Language

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 298–308, Osaka, Japan, December 11-17 2016. Consistent Word Segmentation, Part-of-Speech Tagging and Dependency Labelling Annotation for Chinese Language Mo Shen1, Wingmui Li2, Hyunjeong Choe1, Chenhui Chu3, Daisuke Kawahara4, Sadao Kurohashi4 1 Google Inc., California, USA 2 The Chinese University of Hong Kong, Shatin, Hong Kong 3 Japan Science and Technology Agency 4 Graduate School of Informatics, Kyoto University, Kyoto, Japan {moshen|wjli|hyunjeongc}@google.com, chu@pa.jst.jp, {dk|kuro}@i.kyoto-u.ac.jp Abstract In this paper, we propose a new annotation approach to Chinese word segmentation, part-of- speech (POS) tagging and dependency labelling that aims to overcome the two major issues in traditional morphology-based annotation: Inconsistency and data sparsity. We re-annotate the Penn Chinese Treebank 5.0 (CTB5) and demonstrate the advantages of this approach compared to the original CTB5 annotation through word segmentation, POS tagging and machine transla- tion experiments. 1 Introduction The definition of “word” is an open problem in Chinese linguistics. In previous studies of Chinese cor- pus annotation (Duan et al., 2003; Huang et al., 1997; Xia, 2000), the judgement of word-hood of a meaningful string is based on the analysis of morphology: A morpheme in Chinese is defined as the smallest combination of meaning and phonetic sound in Chinese language, which can be classified into two major types: 1). Free morphemes, which can either be words by themselves or form words with other morphemes; and 2). Bound morphemes, which can only form words by attaching to other morphemes. An issue with word definition using morpheme classification is that, it potentially undermines the consistency of the representation of words. For example, “论” (theory) is a bound morpheme, therefore the string “进化论” (theory of evolution) is treated as a word; on the other hand the string “进化 | 理论” (theory of evolution) are treated as two words, despite the fact that the two strings have the same mean- ing and structure. In another example, “者” (person) is considered as a bound morpheme, therefore “反 对自由贸易者” (people who are against free trade) is treated as one word, while the string without the bound morpheme, i.e. “反对 | 自由 | 贸易” (be against free trade), can only be treated as a phrase of three words. The morphology-based word definition can also make the data sparsity problem worse in corpus an- notation. As an evidence, in the Penn Chinese Treebank 5.0 (CTB5) which is an annotated corpus widely used to train Chinese morphological analysis systems, we found that one of the major sources of the out-of-vocabulary (OOV) words is the compounds that end with a monosyllabic bound morpheme. For example, compounds 利用率 (utility rate) and 次品率 (rate of defective product) end with the bound morpheme 率 (rate); 完成度 (degree of completion) and 活跃度 (degree of activity) end with the bound morpheme 度 (degree); 持续性 (sustainability) and 挥发性 (property of volatile) end with the bound morpheme 性 (property). While these compounds are sparse in the corpus, the morphemes which they This work is licensed under a Creative …

Content truncated to 500 words