Feature-based Neural Language Model and Chinese Word Segmentation

International Joint Conference on Natural Language Processing, pages 1271–1277, Nagoya, Japan, 14-18 October 2013. Feature-based Neural Language Model and Chinese Word Segmentation Mairgup Mansur, Wenzhe Pei, Baobao Chang Key Laboratory of Computational Linguistics, Ministry of Education Institute of Computational Linguistics, Peking University mairgup@gmail.com, williampei1988@126.com, chbb@pku.edu.cn Abstract In this paper we introduce a feature-based neural language model, which is trained to estimate the probability of an element given its previous context features. In this way our feature-based language model can learn representation for more sophisticated features. We introduced the deep neural architecture into the Chinese Word Seg- mentation task. We got a significant im- provement on segmenting performance by sharing the pre-learned representation of character features. The experimental re- sult shows that, while using the same fea- ture sets, our neural segmentation model has a better segmenting performance than CRF-based segmentation model. 1 Introduction Nowadays, as a distributed representation learn- ing framework, Deep Learning Architecture has received a lot of attention in the NLP literature. As Hinton introduced the idea of distributed representation for symbolic data in (Hinton, 1986), this idea has been a research hot spot for more than twenty years. Bengio (2003) applied Hinton’s idea in the con- text of language modeling, and developed a neu- ral language model. Mikolov (2011) improved Bengio’s neural language model by adding recur- rence to the hidden layers, allowing it to beat the state-of-the-art n-gram model not only in terms of perplexity but also in terms of word error rate in speech recognition. Schwenk (2012) applied sim- ilar models in statistical machine translation and improved the BLEU score by almost 2 points. Different from the traditional n-gram model, a distributed representation of word can be learned by neural language model from unlabeled raw data. As the most important philosophy of Deep Learning architecture, the property of learning dis- tributed word representation of neural language model have been proved very useful in NLP tasks. Colobert et al. (2011) developed the SENNA system that shares word representations across the tasks of language modeling, part-of-speech tag- ging, chunking, named entity recognition, seman- tic role labeling and syntactic parsing, which ap- proaches or surpasses the state-of-the-art on these tasks. More interestingly, their experimental result shows that, sharing the word representations that are learned by the neural language model on mas- sive raw data can significantly improve the overall performance of the other tasks. All these research have showed us the very good capability of the neural language model in learn- ing word representation. However, for many NLP tasks, not only the distributed representation of word, but also the representation of features is im- portant. Since the neural language model can learn the representation of words, can we use it to learn representation of features? In this paper, we introduce a more generalized feature-based neural language model, which can learn distributed representation of features, which is very useful for many NLP tasks. Traditional neural language model aims to es- timate the probability of a word given words in …

Content truncated to 500 words