Effect of Non-linear Deep Architecture in Sequence Labeling

International Joint Conference on Natural Language Processing, pages 1285–1291, Nagoya, Japan, 14-18 October 2013. Effect of Non-linear Deep Architecture in Sequence Labeling Mengqiu Wang Computer Science Department Stanford University Stanford, CA 94305, USA mengqiu@cs.stanford.edu Christopher D. Manning Computer Science Department Stanford University Stanford, CA 94305, USA manning@cs.stanford.edu Abstract If we compare the widely used Condi- tional Random Fields (CRF) with newly proposed “deep architecture” sequence models (Collobert et al., 2011), there are two things changing: from linear archi- tecture to non-linear, and from discrete feature representation to distributional. It is unclear, however, what utility non- linearity offers in conventional feature- based models. In this study, we show the close connection between CRF and “se- quence model” neural nets, and present an empirical investigation to compare their performance on two sequence labeling tasks – Named Entity Recognition and Syntactic Chunking. Our results suggest that non-linear models are highly effective in low-dimensional distributional spaces. Somewhat surprisingly, we find that a non- linear architecture offers no benefits in a high-dimensional discrete feature space. 1 Introduction Sequence labeling encompasses an important class of NLP problems that aim at annotating natu- ral language texts with various syntactic and se- mantic information, such as part-of-speech tags and named-entity labels. Output from such sys- tems can facilitate downstream applications such as Question Answering and Relation Extraction. Most methods developed so far for sequence label- ing employ generalized linear statistical models, meaning methods that describe the data as a com- bination of linear basis functions, either directly in the input variables space (e.g., SVM) or through some transformation of the probability distribu- tions (e.g., “log-linear” models). Recently, Collobert et al. (2011) proposed “deep architecture” models for sequence labeling (named Sentence-level Likelihood Neural Nets, abbreviated as SLNN henceforth), and showed promising results on a range of tasks (POS tagging, NER, Chunking, and SRL). Two new changes were suggested: extending the model from a linear to non-linear architecture; and re- placing discrete feature representations with dis- tributional feature representations in a continuous space. It has generally been argued that non- linearity between layers is vital to the power of neural models (Bengio, 2009). The relative con- tribution of these changes, however, is unclear, as is the question of whether gains can be made by introducing non-linearity to conventional feature- based models. In this paper, we illustrate the close relationship between CRF and SLNN models, and conduct an empirical investigation of the effect of nonlinear- ity with different feature representations. Exper- iments on Named Entity Recognition (NER) and Syntactic Chunking tasks suggest that non-linear models are highly effective in low-dimensional distributed feature space, but offer no benefits in high-dimensional discrete space. Furthermore, both linear and non-linear models improve when we combine the discrete and continuous feature spaces, but a linear model still outperforms the non-linear one. 2 From CRFs To SLNNs A CRF models the conditional probability of structured output variables y given observations x. In sequence modeling, the observations are typi- cally words in a sentence, and the output variables …

Content truncated to 500 words