Parsing as Language Modeling

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2331–2336, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Parsing as Language Modeling Do Kook Choe Brown University Providence, RI dc65@cs.brown.edu Eugene Charniak Brown University Providence, RI ec@cs.brown.edu Abstract We recast syntactic parsing as a language modeling problem and use recent advances in neural network language modeling to achieve a new state of the art for constituency Penn Treebank parsing — 93.8 F1 on section 23, us- ing 2-21 as training, 24 as development, plus tri-training. When trees are converted to Stan- ford dependencies, UAS and LAS are 95.9% and 94.1%. 1 Introduction Recent work on deep learning syntactic parsing models has achieved notably good results, e.g., Dyer et al. (2016) with 92.4 F1 on Penn Treebank con- stituency parsing and Vinyals et al. (2015) with 92.8 F1. In this paper we borrow from the ap- proaches of both of these works and present a neural-net parse reranker that achieves very good re- sults, 93.8 F1, with a comparatively simple architec- ture. In the remainder of this section we outline the ma- jor difference between this and previous work — viewing parsing as a language modeling problem. Section 2 looks more closely at three of the most relevant previous papers. We then describe our ex- act model (Section 3), followed by the experimental setup and results (Sections 4 and 5). (a) S VP NP cats chase NP dogs (b) (S (NP dogs )NP (VP chase (NP cats )NP )VP )S Figure 1: A tree (a) and its sequential form (b). There is a one-to-one mapping between a tree and its sequential form. (Part-of-speech tags are not used.) 1.1 Language Modeling Formally, a language model (LM) is a probability distribution over strings of a language: P(x) = P(x1, · · · , xn) = n � t=1 P(xt|x1, · · · , xt−1), (1) where x is a sentence and t indicates a word posi- tion. The efforts in language modeling go into com- puting P(xt|x1, · · · , xt−1), which as described next is useful for parsing as well. 1.2 Parsing as Language Modeling A generative parsing model parses a sentence (x) into its phrasal structure (y) according to argmax y′∈Y(x) P(x, y′), where Y(x) lists all possible structures of x. If we think of a tree (x, y) as a sequence (z) (Vinyals et 2331 al., 2015) as illustrated in Figure 1, we can define a probability distribution over (x, y) as follows: P(x, y) = P(z) = P(z1, · · · , zm) = m � t=1 P(zt|z1, · · · , zt−1), (2) which is equivalent to Equation (1). We have reduced parsing to language modeling and can use language modeling techniques of estimating P(zt|z1, · · · , zt−1) for parsing. 2 Previous Work We look here at three neural net (NN) models clos- est to our research along various dimensions. The first (Zaremba et al., 2014) gives the basic …

Content truncated to 500 words