Neural Network Probability Estimation for Broad Coverage Parsing

Neural Network Probability Estimation for Broad Coverage Parsing James Henderson Departement d'Informatique Universite de Geneve James.Henderson@cui.unige.ch Abstract We present a neural-network-based sta- tistical parser, trained and tested on the Penn Treebank. The neural net- work is used to estimate the parame- ters of a generative model of left-corner parsing, and these parameters are used to search for the most probable parse. The parser's performance (88.8% F- measure) is within 1% of the best cur- rent parsers for this task, despite using a small vocabulary size (512 inputs). Cru- cial to this success is the neural network architecture's ability to induce a finite representation of the unbounded parse history, and the biasing of this induction in a linguistically appropriate way. 1 Introduction Many statistical parsers (Ratnaparkhi, 1999; Collins, 1999; Charniak, 2001) are based on a history-based probability model (Black et al., 1993), where the probability of each decision in a parse is conditioned on the previous decisions in the parse. A major challenge in this approach is choosing a representation of the parse history from which the probability for the next parser decision can be accurately estimated. Previous approaches have used a hand-crafted finite set of features to represent the unbounded parse history (Ratna- parkhi, 1999; Collins, 1999; Charniak, 2001). In the work presented here, we automatically induce a finite set of features to represent the unbounded parse history. We perform this induction using an artificial neural network architecture, called Sim- ple Synchrony Networks (SSNs) (Lane and Hen- derson, 2001; Henderson, 2000). Because this architecture is specifically designed for process- ing structures, it allows us to impose structurally specified and linguistically appropriate biases on the search for a good history representation. The resulting parser achieves performance far greater than previous approaches to neural network pars- ing (Ho and Chan, 1999; Costa et al., 2001), and only marginally below the current state-of-the-art for parsing the Penn Treebank. We propose a hybrid parsing system consist- ing of two components, a neural network which estimates the parameters of a probability model for phrase structure trees, and a statistical parser which searches for the most probable phrase structure tree given these parameters. We first present the probability model which is common to these two components, followed by the estima- tion method, the search method, and a discussion of the empirical results. 2 The Generative Probability Model The probability model we use is generative and history-based. Generative models are expressed in terms of a stochastic process which generates both the phrase structure tree and the input sen- tence. At each step, the process chooses a char- acteristic of the tree or predicts a word in the sen- tence. This sequence of decisions is the derivation of the tree, which we will denote d1,..., dm . Be- cause there is a one-to-one mapping from phrase 131 structure trees to our derivations, the probability of a derivation P(di,..., dm) is equal to the joint probability of the derivation's tree and the input sentence. The probability of the input …

Content truncated to 500 words