Neural Networks Leverage Corpus-wide Information for Part-of-speech Tagging

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 938–950, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Neural Networks Leverage Corpus-wide Information for Part-of-speech Tagging Yuta Tsuboi IBM Research - Tokyo yutat@jp.ibm.com Abstract We propose a neural network approach to benefit from the non-linearity of corpus- wide statistics for part-of-speech (POS) tagging. We investigated several types of corpus-wide information for the words, such as word embeddings and POS tag dis- tributions. Since these statistics are en- coded as dense continuous features, it is not trivial to combine these features com- paring with sparse discrete features. Our tagger is designed as a combination of a linear model for discrete features and a feed-forward neural network that cap- tures the non-linear interactions among the continuous features. By using several re- cent advances in the activation functions for neural networks, the proposed method marks new state-of-the-art accuracies for English POS tagging tasks. 1 Introduction Almost all of the approaches to NLP tasks such as part-of-speech tagging and syntactic parsing mainly use sparse discrete features to represent lo- cal information such as word surfaces in a size- limited window. The non-linearity of those dis- crete features is often used in many NLP tasks since the simple conjunction (AND) of discrete features represents the co-occurrence of the fea- tures and is intuitively understandable. In addi- tion, the thresholding of these combinatorial fea- tures by simple counts effectively suppresses the combinatorial increase of the parameters. At the same time, although global information had also been used in several reports (Nakagawa and Mat- sumoto, 2006; Huang and Yates, 2009; Turian et al., 2010; Schnabel and Sch¨utze, 2014), the non- linear interactions of these features were not well investigated since these features are often dense continuous features and the explicit non-linear ex- pansions are counterintuitive and drastically in- crease the number of the model parameters. In our work, we investigate neural networks used to rep- resent the non-linearity of global information for POS tagging in a compact way. We focus on four kinds of corpus-wide infor- mation: (1) word embeddings, (2) POS tag dis- tributions, (3) supertag distributions, and (4) con- text word distributions. All of them are continuous dense features and we use a feed-forward neural network to exploit the non-linearity of these fea- tures. Although all of them except (3) have been used for POS tagging in previous work (Nakamura et al., 1990; Schmid, 1994; Schnabel and Sch¨utze, 2014; Huang and Yates, 2009), we propose a neu- ral network approach to capture the non-linear in- teractions of these features. By feeding these fea- tures into neural networks as an input vector, we can expect our tagger can handle not only the non- linearity of the N-grams of the same kinds of fea- tures but also the non-linear interactions among the different kind of features. Our tagger combines a linear model using sparse high-dimensional features and a neural net- work using continuous dense features. Although Collobert et al. (2011) …

Content truncated to 500 words