Word-Context Character Embeddings for Chinese Word Segmentation

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 760–766 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Word-Context Character Embeddings for Chinese Word Segmentation Hao Zhou∗ Nanjing University & Toutiao AI Lab zhouhao.nlp@bytedance.com Zhenting Yu∗ Nanjing University yuzt@nlp.nju.edu.cn Yue Zhang Singapore University of Technology and Design yue_zhang@sutd.edu.sg Shujian Huang Nanjing University huangsj@nlp.nju.edu.cn Xinyu Dai Nanjing University daixinyu@nju.edu.cn Jiajun Chen Nanjing University chenjj@nlp.nju.edu.cn Abstract Neural parsers have benefited from au- tomatically labeled data via dependency- context word embeddings. We inves- tigate training character embeddings on a word-based context in a similar way, showing that the simple method signif- icantly improves state-of-the-art neural word segmentation models, beating tri- training baselines for leveraging auto- segmented data. 1 Introduction Neural network Chinese word segmenta- tion (CWS) models (Zhang et al., 2016; Liu et al., 2016; Cai and Zhao, 2016) appeal for their strong ability of feature representation, employing unigram and bigram character embeddings as input features (Zheng et al., 2013; Pei et al., 2014; Ma and Hinrichs, 2015; Chen et al., 2015a). They give state-of-the-art performances. We investigate leveraging automatically segmented texts for enhancing their accuracies. Such semi-supervised methods can be divided into two main categories. The first one is boot- strapping, which includes self-training and tri- training. The idea is to generate more training in- stances by automatically labeling large-scale data. Self-training (Yarowsky, 1995; McClosky et al., 2006; Huang et al., 2010; Liu and Zhang, 2012) labels additional data by using the base classifier itself, and tri-training (Zhou and Li, 2005; Li et al., 2014) uses two extra classifiers, taking the in- stances with the same labels for additional training data. A second semi-supervised learning method in NLP is knowledge distillation, which extracts knowledge from large-scale auto-labeled data as features. ∗Equal contributions Tri-training has been used in neural parsing, giv- ing considerable improvements for both of depen- dency (Weiss et al., 2015) and constituent pars- ing (Vinyals et al., 2015; Choe and Charniak, 2016). Knowledge from auto-labeled data has also been used for parsing (Bansal et al., 2014; Melamud et al., 2016), where word embeddings are trained on automatic dependency tree context. Such knowledge has also been proved effective in conventional discrete CWS models, such as label distribution information (Wang et al., 2011; Zhang et al., 2013). However, it has not been investigated for neural CWS. We propose word-context character embed- dings (WCC), using segmentation label informa- tion in the pre-training of unigram and bigram character embeddings. The method packs the la- bel distribution information into the embeddings, which could be regarded as a way for knowl- edge parameterization. Our idea follows Levy and Goldberg (2014), who use dependency contexts to train word embeddings. Additionally, moti- vated by co-training, we propose multi-view word- context character embeddings for cross-domain segmentation, which pre-trains two types of em- bedding for in-domain and out-of-domain data, re- spectively. In-domain embeddings are used for solving data sparseness, and out-of-domain em- beddings are used for domain adaptation. Our proposed model is simple, efficient …

Content truncated to 500 words