Multiplicative Tree-Structured Long Short-Term Memory Networks for Semantic Representations

Proceedings of the 7th Joint Conference on Lexical and Computational Semantics (*SEM), pages 276–286 New Orleans, June 5-6, 2018. c⃝2018 Association for Computational Linguistics Multiplicative Tree-Structured Long Short-Term Memory Networks for Semantic Representations Nam Khanh Tran∗ L3S Research Center ntran@l3s.de Weiwei Cheng Amazon weiweic@amazon.com Abstract Tree-structured LSTMs have shown advan- tages in learning semantic representations by exploiting syntactic information. Most exist- ing methods model tree structures by bottom- up combinations of constituent nodes using the same shared compositional function and often making use of input word information only. The inability to capture the richness of com- positionality makes these models lack expres- sive power. In this paper, we propose multi- plicative tree-structured LSTMs to tackle this problem. Our model makes use of not only word information but also relation information between words. It is more expressive, as dif- ferent combination functions can be used for each child node. In addition to syntactic trees, we also investigate the use of Abstract Mean- ing Representation in tree-structured models, in order to incorporate both syntactic and se- mantic information from the sentence. Experi- mental results on common NLP tasks show the proposed models lead to better sentence repre- sentation and AMR brings benefits in complex tasks. 1 Introduction Learning the distributed representation for long spans of text from its constituents has been a crucial step of various NLP tasks such as text classification (Zhao et al., 2015; Kim, 2014), se- mantic matching (Liu et al., 2016), and machine translation (Cho et al., 2014). Seminal work uses recurrent neural networks (RNN) (Elman, 1990), convolutional neural networks (Kalchbren- ner et al., 2014), and tree-structured neural net- works (Socher et al., 2011; Tai et al., 2015) for sequence and tree modeling. Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) networks are a type of recurrent neural net- ∗ Work done as an intern at Amazon. 𝑐"#$ ℎ"#$ 𝑐" ℎ" 𝑥"#$ 𝑥" 𝑐"#$ ' ℎ"#$ ' 𝑐"#$ ( ℎ"#$ ( 𝑐" ℎ" 𝑥" 𝑥"#$ ' 𝑥"#$ ( (a) (b) Figure 1: Topology of sequential LSTM and TreeLSTM: (a) nodes in sequential LSTM and (b) nodes in tree-structured LSTM work that are capable of learning long-term de- pendencies across sequences and have achieved significant improvements in a variety of sequence tasks. LSTM has been extended to model tree structures (e.g., TreeLSTM) and produced promis- ing results in tasks such as sentiment classification (Tai et al., 2015; Zhu et al., 2015) and relation ex- traction (Miwa and Bansal, 2016). Figure 1 shows the topologies of the con- ventional chain-structured LSTM (Hochreiter and Schmidhuber, 1997) and the TreeLSTM (Tai et al., 2015), illustrating the input (x), cell (c) and hid- den node (h) at a time step t. The key difference between Figure 1 (a) and (b) is the branching fac- tor. While a cell in the sequential LSTM only de- pends on the single previous hidden node, a cell in the tree-structured LSTM depends on the hidden states of child nodes. Despite their success, the tree-structured mod- els have …

Content truncated to 500 words