Top-down Tree Long Short-Term Memory Networks

Proceedings of NAACL-HLT 2016, pages 310–320, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Top-down Tree Long Short-Term Memory Networks Xingxing Zhang, Liang Lu and Mirella Lapata School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB, UK {x.zhang,liang.lu}@ed.ac.uk,mlap@inf.ed.ac.uk Abstract Long Short-Term Memory (LSTM) networks, a type of recurrent neural network with a more complex computational unit, have been successfully applied to a variety of sequence modeling tasks. In this paper we develop Tree Long Short-Term Memory (TREELSTM), a neural network model based on LSTM, which is designed to predict a tree rather than a lin- ear sequence. TREELSTM defines the prob- ability of a sentence by estimating the gener- ation probability of its dependency tree. At each time step, a node is generated based on the representation of the generated sub- tree. We further enhance the modeling power of TREELSTM by explicitly representing the correlations between left and right depen- dents. Application of our model to the MSR sentence completion challenge achieves re- sults beyond the current state of the art. We also report results on dependency parsing reranking achieving competitive performance. 1 Introduction Neural language models have been gaining increas- ing attention as a competitive alternative to n-grams. The main idea is to represent each word using a real-valued feature vector capturing the contexts in which it occurs. The conditional probability of the next word is then modeled as a smooth function of the feature vectors of the preceding words and the next word. In essence, similar representations are learned for words found in similar contexts result- ing in similar predictions for the next word. Previ- ous approaches have mainly employed feed-forward (Bengio et al., 2003; Mnih and Hinton, 2007) and recurrent neural networks (Mikolov et al., 2010; Mikolov, 2012) in order to map the feature vec- tors of the context words to the distribution for the next word. Recently, RNNs with Long Short-Term Memory (LSTM) units (Hochreiter and Schmidhu- ber, 1997; Hochreiter, 1998) have emerged as a pop- ular architecture due to their strong ability to capture long-term dependencies. LSTMs have been success- fully applied to a variety of tasks ranging from ma- chine translation (Sutskever et al., 2014), to speech recognition (Graves et al., 2013), and image descrip- tion generation (Vinyals et al., 2015). Despite superior performance in many applica- tions, neural language models essentially predict se- quences of words. Many NLP tasks, however, ex- ploit syntactic information operating over tree struc- tures (e.g., dependency or constituent trees). In this paper we develop a novel neural network model which combines the advantages of the LSTM archi- tecture and syntactic structure. Our model estimates the probability of a sentence by estimating the gen- eration probability of its dependency tree. Instead of explicitly encoding tree structure as a set of fea- tures, we use four LSTM networks to model four types of dependency edges which altogether specify how the tree is built. At each time step, one LSTM is activated which predicts the …

Content truncated to 500 words