Deep Learning for Chinese Word Segmentation and POS Tagging

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 647–657, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Deep Learning for Chinese Word Segmentation and POS Tagging Xiaoqing Zheng Fudan University 220 Handan Road Shanghai, 200433, China zhengxq@fudan.edu.cn Hanyang Chen Fudan University 220 Handan Road Shanghai, 200433, China chenhy12345@gmail.com Tianyu Xu Fudan University 220 Handan Road Shanghai, 200433, China xty213@gmail.com Abstract This study explores the feasibility of perform- ing Chinese word segmentation (CWS) and POS tagging by deep learning. We try to avoid task-specific feature engineering, and use deep layers of neural networks to discover relevant features to the tasks. We leverage large-scale unlabeled data to improve internal representa- tion of Chinese characters, and use these im- proved representations to enhance supervised word segmentation and POS tagging models. Our networks achieved close to state-of-the- art performance with minimal computational cost. We also describe a perceptron-style al- gorithm for training the neural networks, as an alternative to maximum-likelihood method, to speed up the training process and make the learning algorithm easier to be implemented. 1 Introduction Word segmentation has been a long-standing chal- lenge for the Chinese NLP community. It has re- ceived steady attention over the past two decades. Previous studies show that joint solutions usually lead to the improvement in accuracy over pipelined systems by exploiting POS information to help word segmentation and avoiding error propagation. How- ever, traditional joint approaches usually involve a great number of features, which arises four limita- tions. First, the size of the result models is too large for practical use due to the storage and computing constraints of certain real-world applications. Sec- ond, the number of parameters is so large that the trained model is apt to overfit on training corpus. Third, a longer training time is required. Last but not the least, the decoding by dynamic programming technique might be intractable since a large search space is faced by the decoder. The choice of features, therefore, is a critical suc- cess factor for these systems. Most of the state-of- the-art systems address their tasks by applying linear statistical models to the features carefully optimized for the tasks. This approach is effective because re- searchers can incorporate a large body of linguistic knowledge into the models. However, the approach does not scale well when it is used to perform more complex joint tasks, for example, the task of joint word segmentation, POS tagging, parsing, and se- mantic role labeling. A challenge for such a joint model is the large combined search space, which makes engineering effective task-specific features and structured learning of parameters very hard. In- stead, we use multilayer neural networks to discover the useful features from the input sentences. There are two main contributions in this paper. (1) We describe a perceptron-style algorithm for train- ing the neural networks, which not only speeds up the training of the networks with negligible loss in performance, but also can be implemented more eas- ily; (2) We …

Content truncated to 500 words