Multi-Grained Chinese Word Segmentation

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 692–703 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Multi-Grained Chinese Word Segmentation Chen Gong, Zhenghua Li∗, Min Zhang, Xinzhou Jiang Soochow University, Suzhou, China cgong@stu.suda.edu.cn, {zhli13,minzhang}@suda.edu.cn, xzjiang.hw@gmail.com Abstract Traditionally, word segmentation (WS) adopts the single-granularity formal- ism, where a sentence corresponds to a single word sequence. However, Sproat et al. (1996) show that the inter-native- speaker consistency ratio over Chinese word boundaries is only 76%, indicat- ing single-grained WS (SWS) imposes unnecessary challenges on both manual annotation and statistical modeling. Moreover, WS results of different gran- ularities can be complementary and beneficial for high-level applications. This work proposes and addresses multi-grained WS (MWS). First, we build a large-scale pseudo MWS dataset for model training and tuning by leveraging the annotation heterogeneity of three SWS datasets. Then we manually annotate 1,500 test sentences with true MWS annotations. Finally, we propose three benchmark approaches by casting MWS as constituent parsing and sequence labeling. Experiments and analysis lead to many interesting findings. 1 Introduction As the first processing step of Chinese language processing, word segmentation (WS) has been extensively studied and made great progress during the past decades, thanks to the annotation of large-scale benchmark datasets, among which the most widely- used are Microsoft Research Corpus (MSR) (Huang et al., 2006), Peking University ∗ Correspondence author MSR 全国各地 医学 界 专家 走出 人民大会堂 PPD 全国 各地 医学界 专家 走 出 人民 大会堂 CTB 全 国 各 地 医学界 专家 走出 人民 大会堂 Table 1: An example of annotation heterogeneity: 全 (all) 国 (country) 各 (every) 地 (place) 医学 (medical science) 界 (field) 专 家 (experts) 走 (walk) 出 (out) 人民 (people) 大会堂 (great hall). People Daily Corpus (PPD) (Yu et al., 2003), and Penn Chinese Treebank (CTB) (Xue et al., 2005). Table 1 gives an example sentence segmented in different guidelines. Meanwhile, WS approaches gradually evolve from maximum matching based on lexicon dictionaries (Liu and Liang, 1986), to path searching from segmentation graphs based on language modeling scores and other statistics (Zhang and Liu, 2002), to character-based sequence labeling (Xue, 2003), to shift-reduce incremental parsing (Zhang and Clark, 2007). Recently, neural network models have also achieved success by effectively learning representation of characters and contexts (Zheng et al., 2013; Pei et al., 2014; Ma and Hinrichs, 2015; Chen et al., 2015; Zhang et al., 2016; Cai and Zhao, 2016; Liu et al., 2016). To date, all the labeled datasets adopt the single-granularity formalization, and previous research mainly focuses on single-grained WS (SWS), where one sentence is segmented into a single word sequence. Although different WS guidelines share the same high-level criterion of word boundaries – a character string com- bined closely and used steadily forms a word, people greatly diverge due to individual differ- ences on knowledge and living environments, etc. An anonymous reviewer kindly points 692 out that Vladímir Skalička of the Prague School claimed that unlike the “isolating” languages such as French and English, Chinese …

Content truncated to 500 words