Neural Syntactic Generative Models with Exact Marginalization

Proceedings of NAACL-HLT 2018, pages 942–952 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Neural Syntactic Generative Models with Exact Marginalization Jan Buys1,2 and Phil Blunsom1,3 1Department of Computer Science, University of Oxford 2Paul G. Allen School of Computer Science & Engineering, University of Washington 3DeepMind jbuys@cs.washington.edu, phil.blunsom@cs.ox.ac.uk Abstract We present neural syntactic generative mod- els with exact marginalization that support both dependency parsing and language mod- eling. Exact marginalization is made tractable through dynamic programming over shift- reduce parsing and minimal RNN-based fea- ture sets. Our algorithms complement previ- ous approaches by supporting batched train- ing and enabling online computation of next word probabilities. For supervised depen- dency parsing, our model achieves a state- of-the-art result among generative approaches. We also report empirical results on unsuper- vised syntactic models and their role in lan- guage modeling. We find that our model for- mulation of latent dependencies with exact marginalization do not lead to better intrinsic language modeling performance than vanilla RNNs, and that parsing accuracy is not cor- related with language modeling perplexity in stack-based models. 1 Introduction We investigate the feasibility of neural syntactic generative models with structured latent variables in which exact inference is tractable. Recent mod- els have added structure to recurrent neural net- works at the cost of giving up exact inference, or through using soft structure instead of latent vari- ables (Dyer et al., 2016; Yogatama et al., 2016; Grefenstette et al., 2015). We propose genera- tive models in which syntactic structure is mod- elled with a discrete stack which can be marginal- ized as a latent variable through dynamic program- ming. This enables us to investigate the trade-off between model expressivity and exact marginal- ization in probabilistic models based on recurrent neural networks (RNNs). While Long Short-term Memory (Hochreiter and Schmidhuber, 1997) (LSTM) RNNs have driven strong improvements in intrinsic language modelling performance, they fail at capturing cer- tain long-distance dependencies, such as those required for modelling subject-verb agreement (Linzen et al., 2016) or performing synthetic trans- duction tasks based on context-free grammars (Grefenstette et al., 2015). We propose genera- tive models, based on transition-based dependency parsing (Nivre, 2008), a widely used framework for incremental syntactic parsing, that are able to capture desirable dependencies. Our generative approach to dependency parsing encodes sentences with an RNN and estimate tran- sition and next word probability distributions by conditioning on a small number of features repre- sented by RNN encoder vectors. In contrast to pre- vious syntactic language models such as RNNG (Dyer et al., 2016), marginal word probabilities can be computed both online and exactly. A GPU implementation which exploits parallelization en- ables unsupervised learning and fast training and decoding. The price of exact inference is that our models are less expressive than RNNG, as the re- currence is not syntax-dependent. Our generative models are based on the arc- eager and arc-hybrid transition systems, with O(n3) dynamic programs based on Kuhlmann et al. (2011). Previous work on dynamic pro- gramming …

Content truncated to 500 words