Head-Lexicalized Bidirectional Tree LSTMs

Head-Lexicalized Bidirectional Tree LSTMs Zhiyang Teng and Yue Zhang Singapore University of Technology and Design zhiyang teng@mymail.sutd.edu.sg yue zhang@sutd.edu.sg Abstract Sequential LSTMs have been extended to model tree structures, giving competitive re- sults for a number of tasks. Existing methods model constituent trees by bottom-up com- binations of constituent nodes, making di- rect use of input word information only for leaf nodes. This is different from sequen- tial LSTMs, which contain references to input words for each node. In this paper, we propose a method for automatic head-lexicalization for tree-structure LSTMs, propagating head words from leaf nodes to every constituent node. In addition, enabled by head lexicaliza- tion, we build a tree LSTM in the top-down direction, which corresponds to bidirectional sequential LSTMs in structure. Experiments show that both extensions give better repre- sentations of tree structures. Our final model gives the best results on the Stanford Senti- ment Treebank and highly competitive results on the TREC question type classification task. 1 Introduction Both sequence structured and tree structured neural models have been applied to NLP problems. Sem- inal work uses convolutional neural networks (Col- lobert and Weston, 2008), recurrent neural networks (Elman, 1990; Mikolov et al., 2010) and recur- sive neural networks (Socher et al., 2011) for se- quence and tree modeling. Long short-term mem- ory (LSTM) networks have significantly improved accuracies in a variety of sequence tasks (Sutskever et al., 2014; Bahdanau et al., 2015) compared to vanilla recurrent neural networks. Addressing di- minishing gradients effectively, they have been ex- tended to tree structures, achieving promising re- sults for tasks such as syntactic language modeling (Zhang et al., 2016), sentiment analysis (Li et al., 2015; Zhu et al., 2015; Le and Zuidema, 2015; Tai et al., 2015; Teng et al., 2016) and relation extraction (Miwa and Bansal, 2016). Depending on the node type, typical tree struc- tures in NLP can be categorized to constituent trees and dependency trees. A salient difference be- tween the two types of tree structures is in the node. While dependency tree nodes are input words themselves, constituent tree nodes represent syntac- tic constituents. Only leaf nodes in constituent trees correspond to words. Though LSTM structures have been developed for both types of trees above, we in- vestigate constituent trees in this paper. There are three existing methods for constituent tree LSTMs (Zhu et al., 2015; Tai et al., 2015; Le and Zuidema, 2015), which make essentially the same extension from sequence structured LSTMs. We take the method of Zhu et al. (2015) as our baseline. Figure 1 shows the sequence structured LSTM of Hochreiter and Schmidhuber (1997) and the tree- structured LSTM of Zhu et al. (2015), illustrating the input (x), cell (c) and hidden (h) nodes at a cer- tain time step t. The most important difference be- tween Figure 1(a) and Figure 1(b) is the branching factor. While a cell in the sequence structure LSTM depends on the single previous hidden node, a cell in the tree-structured LSTM depends on a …

Content truncated to 500 words