Neural CRF Parsing

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 302–312, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Neural CRF Parsing Greg Durrett and Dan Klein Computer Science Division University of California, Berkeley {gdurrett,klein}@cs.berkeley.edu Abstract This paper describes a parsing model that combines the exact dynamic programming of CRF parsing with the rich nonlinear fea- turization of neural net approaches. Our model is structurally a CRF that factors over anchored rule productions, but in- stead of linear potential functions based on sparse features, we use nonlinear po- tentials computed via a feedforward neu- ral network. Because potentials are still local to anchored rules, structured infer- ence (CKY) is unchanged from the sparse case. Computing gradients during learn- ing involves backpropagating an error sig- nal formed from standard CRF sufficient statistics (expected rule counts). Us- ing only dense features, our neural CRF already exceeds a strong baseline CRF model (Hall et al., 2014). In combination with sparse features, our system1 achieves 91.1 F1 on section 23 of the Penn Tree- bank, and more generally outperforms the best prior single parser results on a range of languages. 1 Introduction Neural network-based approaches to structured NLP tasks have both strengths and weaknesses when compared to more conventional models, such conditional random fields (CRFs). A key strength of neural approaches is their ability to learn nonlinear interactions between underlying features. In the case of unstructured output spaces, this capability has led to gains in problems rang- ing from syntax (Chen and Manning, 2014; Be- linkov et al., 2014) to lexical semantics (Kalch- brenner et al., 2014; Kim, 2014). Neural methods are also powerful tools in the case of structured 1System available at http://nlp.cs.berkeley.edu output spaces. Here, past work has often relied on recurrent architectures (Henderson, 2003; Socher et al., 2013; ˙Irsoy and Cardie, 2014), which can propagate information through structure via real- valued hidden state, but as a result do not admit ef- ficient dynamic programming (Socher et al., 2013; Le and Zuidema, 2014). However, there is a nat- ural marriage of nonlinear induced features and efficient structured inference, as explored by Col- lobert et al. (2011) for the case of sequence mod- eling: feedforward neural networks can be used to score local decisions which are then “reconciled” in a discrete structured modeling framework, al- lowing inference via dynamic programming. In this work, we present a CRF constituency parser based on these principles, where individ- ual anchored rule productions are scored based on nonlinear features computed with a feedfor- ward neural network. A separate, identically- parameterized replicate of the network exists for each possible span and split point. As input, it takes vector representations of words at the split point and span boundaries; it then outputs scores for anchored rules applied to that span and split point. These scores can be thought of as non- linear potentials analogous to linear potentials in conventional CRFs. Crucially, while the network …

Content truncated to 500 words