Comparing Top-Down and Bottom-Up Neural Generative Dependency Models

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 227–237 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 227 Comparing Top-down and Bottom-up Neural Generative Dependency Models Austin Matthews and Graham Neubig Language Technologies Institute Carnegie Mellon University {austinma,gneubig}@cs.cmu.edu Chris Dyer DeepMind cdyer@google.com Abstract Recurrent neural network grammars (RNNGs) generate sentences using phrase-structure syn- tax and perform very well in terms of both language modeling and parsing performance. However, since dependency annotations are much more readily available than phrase struc- ture annotations, we propose two new gen- erative models of projective dependency syn- tax, so as to explore whether generative de- pendency models are similarly effective. Both models use RNNs to represent the deriva- tion history with making any explicit indepen- dence assumptions, but they differ in how they construct the trees: one builds the tree bot- tom up and the other top down, which pro- foundly changes the estimation problem faced by the learner. We evaluate the two mod- els on three typologically different languages: English, Arabic, and Japanese. We find that both generative models improve parsing per- formance over a discriminative baseline, but, in contrast to RNNGs, they are significantly less effective than non-syntactic LSTM lan- guage models. Little difference between the tree construction orders is observed for either parsing or language modeling. 1 Introduction Recurrent neural network grammars (Dyer et al., 2016, RNNGs) are syntactic language models that use predicted syntactic structures to determine the topology of the recurrent networks they use to pre- dict subsequent words. Not only can they learn to model language better than non-syntactic lan- guage models, but the conditional distributions over parse trees given sentences produce excellent parsers (Fried et al., 2017). In this paper, we introduce and evaluate two new dependency syntax language models which are based on a recurrent neural network (RNN) backbone (§2).1 Dependency syntax is particu- larly appealing as many more languages have de- pendency treebanks (e.g. the Universal Dependen- cies Project (Nivre et al., 2017)) than have large numbers of phrase structure annotations. Like RNNGs, our proposed models predict structure and words jointly, and the predicted syn- tactic structure is used to determine the structure of the neural network that is used to represent the history of actions taken by the model and to make a better estimate of the distribution over subse- quent structure-building and word-generating ac- tions. Because we use RNNs to encode the deriva- tion history, our models do not make any explicit independence assumptions, but instead condition on the complete history of actions. The two pro- posed models do, however, differ in the order that they construct the trees. The first model operates top down (§2.1), starting at the root and recur- sively generating dependents until the last modi- fier has been generated. The second operates bot- tom up (§2.2), generating words from left to right and interleaving decisions about how they fit to- gether to form tree fragments and finally a fully formed dependency tree.2 Because neither …

Content truncated to 500 words