Human-Computer Interactive Chinese Word Segmentation: An Adaptive Dirichlet Process Mixture Model Approach

International Joint Conference on Natural Language Processing, pages 1278–1284, Nagoya, Japan, 14-18 October 2013. Human-Computer Interactive Chinese Word Segmentation: An Adaptive Dirichlet Process Mixture Model Approach Tongfei Chen1, Xiaojun Zou2, Weimeng Zhu1, Junfeng Hu2, * 1 School of Electronics Engineering and Computer Science, Peking University, Beijing, 100871, P. R. China 2 Key Laboratory of Computational Linguistics (Peking University), Ministry of Education, Beijing, 100871, P. R. China {ctf,zouxj,zwm,hujf}@pku.edu.cn Abstract Previous research shows that Kalman fil- ter based human-computer interactive Chinese word segmentation achieves an encouraging effect in reducing user inter- ventions, but suffers from the drawback of incompetence in distinguishing seg- mentation ambiguities. This paper pro- poses a novel approach to handle this problem by using an adaptive Dirichlet process mixture model. By adjusting the hyperparameters of the model, ideal clas- sifiers can be generated to conform to the interventions provided by the users. Ex- periments reveal that our approach achieves a notable improvement in han- dling segmentation ambiguities. With knowledge learnt from users, our model outperforms the baseline Kalman filter model by about 0.5% in segmenting ho- mogeneous texts. 1 Introduction * As Chinese text is written without natural delim- iters such as whitespaces, word segmentation is often the essential first step in Chinese language processing (Liang, 1987). Over the past two dec- ades, various methods have been developed to address this issue (Nie et al., 1994; Sun et al., 1998; Luo et al., 2002; Zhang et al., 2003; Peng et al., 2004; Goldwater et al., 2006). Generally, supervised statistical learning methods are more adaptive and robust in processing unrestricted texts than the traditional dictionary-based meth- ods. * To whom all correspondence should be addressed. However, in some domain-specific applica- tions, for example ancient Chinese text pro- cessing, there is neither enough homogeneous corpora for training a reliable statistical model, nor a well-defined dictionary. In these tasks, un- supervised word segmentation is preferred to utilize the linguistic knowledge derived from the raw corpus itself. Many researches also enable users to take part in the segmentation process, adding expert knowledge to the system (Wang et al., 2002; Li and Chen, 2007). This is quite rea- sonable since the criteria of word segmentation are dependent on a user or the destination of use in many applications (Sproat et al., 1996). Zhu et al. (2013) proposed a Kalman filter based human-computer interactive learning mod- el for segmenting Chinese texts depending upon neither lexicon nor any annotated corpus. This approach enables experts to observe and inter- vene with the segmentation results, while the segmenter learns and adapts to these knowledge iteratively. At the end of this procedure, a seg- mentation result that fully matches the demand of the user is returned. However, in some com- plicated cases where segmentation ambiguities exist, the Kalman filter will not converge and keep swapping in two or more states. To overcome this drawback, we established an adaptive Dirichlet process mixture model (ADPMM) for human-computer interactive word segmentation. ADPMM gradually adapts itself to the knowledge supplied by users through …

Content truncated to 500 words