Effective Inference for Generative Neural Parsing

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1695–1700 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Effective Inference for Generative Neural Parsing Mitchell Stern Daniel Fried Computer Science DivisionDan Klein University of California, Berkeley {mitchell,dfried,klein}@cs.berkeley.edu Abstract Generative neural models have recently achieved state-of-the-art results for con- stituency parsing. However, without a fea- sible search procedure, their use has so far been limited to reranking the output of ex- ternal parsers in which decoding is more tractable. We describe an alternative to the conventional action-level beam search used for discriminative neural models that enables us to decode directly in these gen- erative models. We then show that by im- proving our basic candidate selection strat- egy and using a coarse pruning function, we can improve accuracy while explor- ing significantly less of the search space. Applied to the model of Choe and Char- niak (2016), our inference procedure ob- tains 92.56 F1 on section 23 of the Penn Treebank, surpassing prior state-of-the-art results for single-model systems. 1 Introduction A recent line of work has demonstrated the success of generative neural models for constituency pars- ing (Dyer et al., 2016; Choe and Charniak, 2016). As with discriminative neural parsers, these mod- els lack a dynamic program for exact inference due to their modeling of unbounded dependencies. However, while discriminative neural parsers are able to obtain strong results using greedy search (Dyer et al., 2016) or beam search with a small beam (Vinyals et al., 2015), we find that a simple action-level approach fails outright in the genera- tive setting. Perhaps because of this, the applica- tion of generative neural models has so far been re- stricted to reranking the output of external parsers. Intuitively, because a generative parser defines a joint distribution over sentences and parse trees, probability mass will be allocated unevenly be- tween a small number of common structural ac- tions and a large vocabulary of lexical items. This imbalance is a primary cause of failure for search procedures in which these two types of ac- tions compete directly. A notion of equal com- petition among hypotheses is then desirable, an idea that has previously been explored in gener- ative models for constituency parsing (Henderson, 2003) and dependency parsing (Titov and Hen- derson, 2010; Buys and Blunsom, 2015), among other tasks. We describe a related state-augmented beam search for neural generative constituency parsers in which lexical actions compete only with each other rather than with structural actions. Ap- plying this inference procedure to the generative model of Choe and Charniak (2016), we find that it yields a self-contained generative parser that achieves high performance. Beyond this, we propose an enhanced candi- date selection strategy that yields significant im- provements for all beam sizes. Additionally, mo- tivated by the look-ahead heuristic used in the top-down parsers of Roark (2001) and Charniak (2010), we also experiment with a simple coarse pruning function that allows us to reduce the num- ber of states expanded per …

Content truncated to 500 words