Discriminative Training of a Neural Network Statistical Parser

Discriminative Training of a Neural Network Statistical Parser James HENDERSON School of Informatics, University of Edinburgh 2 Buccleuch Place Edinburgh EH8 9LW United Kingdom james.henderson@ed.ac.uk Abstract Discriminative methods have shown significant improvements over traditional generative meth- ods in many machine learning applications, but there has been difficulty in extending them to natural language parsing. One problem is that much of the work on discriminative methods conflates changes to the learning method with changes to the parameterization of the problem. We show how a parser can be trained with a dis- criminative learning method while still param- eterizing the problem according to a generative probability model. We present three methods for training a neural network to estimate the probabilities for a statistical parser, one gen- erative, one discriminative, and one where the probability model is generative but the training criteria is discriminative. The latter model out- performs the previous two, achieving state-of- the-art levels of performance (90.1% F-measure on constituents). 1 Introduction Much recent work has investigated the applica- tion of discriminative methods to NLP tasks, with mixed results. Klein and Manning (2002) argue that these results show a pattern where discriminative probability models are inferior to generative probability models, but that im- provements can be achieved by keeping a gener- ative probability model and training according to a discriminative optimization criteria. We show how this approach can be applied to broad coverage natural language parsing. Our estima- tion and training methods successfully balance the conflicting requirements that the training method be both computationally tractable for large datasets and a good approximation to the theoretically optimal method. The parser which uses this approach outperforms both a genera- tive model and a discriminative model, achiev- ing state-of-the-art levels of performance (90.1% F-measure on constituents). To compare these different approaches, we use a neural network architecture called Sim- ple Synchrony Networks (SSNs) (Lane and Hen- derson, 2001) to estimate the parameters of the probability models. SSNs have the advantage that they avoid the need to impose hand-crafted independence assumptions on the learning pro- cess. Training an SSN simultaneously trains a finite representations of the unbounded parse history and a mapping from this history repre- sentation to the parameter estimates. The his- tory representations are automatically tuned to optimize the parameter estimates. This avoids the problem that any choice of hand-crafted in- dependence assumptions may bias our results towards one approach or another. The indepen- dence assumptions would have to be different for the generative and discriminative probabil- ity models, and even for the parsers which use the generative probability model, the same set of independence assumptions may be more ap- propriate for maximizing one training criteria over another. By inducing the history represen- tations specifically to fit the chosen model and training criteria, we avoid having to choose in- dependence assumptions which might bias our results. Each complete parsing system we propose consists of three components, a probability model for sequences of parser decisions, a Sim- ple Synchrony Network which estimates the pa- …

Content truncated to 500 words