Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1556–1566, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks Kai Sheng Tai, Richard Socher*, Christopher D. Manning Computer Science Department, Stanford University, *MetaMind Inc. kst@cs.stanford.edu, richard@metamind.io, manning@stanford.edu Abstract Because of their superior ability to pre- serve sequence information over time, Long Short-Term Memory (LSTM) net- works, a type of recurrent neural net- work with a more complex computational unit, have obtained strong results on a va- riety of sequence modeling tasks. The only underlying LSTM structure that has been explored so far is a linear chain. However, natural language exhibits syn- tactic properties that would naturally com- bine words to phrases. We introduce the Tree-LSTM, a generalization of LSTMs to tree-structured network topologies. Tree- LSTMs outperform all existing systems and strong LSTM baselines on two tasks: predicting the semantic relatedness of two sentences (SemEval 2014, Task 1) and sentiment classification (Stanford Senti- ment Treebank). 1 Introduction Most models for distributed representations of phrases and sentences—that is, models where real- valued vectors are used to represent meaning—fall into one of three classes: bag-of-words models, sequence models, and tree-structured models. In bag-of-words models, phrase and sentence repre- sentations are independent of word order; for ex- ample, they can be generated by averaging con- stituent word representations (Landauer and Du- mais, 1997; Foltz et al., 1998). In contrast, se- quence models construct sentence representations as an order-sensitive function of the sequence of tokens (Elman, 1990; Mikolov, 2012). Lastly, tree-structured models compose each phrase and sentence representation from its constituent sub- phrases according to a given syntactic structure over the sentence (Goller and Kuchler, 1996; Socher et al., 2011). x1 x2 x3 x4 y1 y2 y3 y4 x1 x2 x4 x5 x6 y1 y2 y3 y4 y6 Figure 1: Top: A chain-structured LSTM net- work. Bottom: A tree-structured LSTM network with arbitrary branching factor. Order-insensitive models are insufficient to fully capture the semantics of natural language due to their inability to account for differences in meaning as a result of differences in word order or syntactic structure (e.g., “cats climb trees” vs. “trees climb cats”). We therefore turn to order- sensitive sequential or tree-structured models. In particular, tree-structured models are a linguisti- cally attractive option due to their relation to syn- tactic interpretations of sentence structure. A nat- ural question, then, is the following: to what ex- tent (if at all) can we do better with tree-structured models as opposed to sequential models for sen- tence representation? In this paper, we work to- wards addressing this question by directly com- paring a type of sequential model that has recently been used to achieve state-of-the-art results in sev- eral NLP tasks against its tree-structured general- ization. Due to their capability for processing arbitrary- length sequences, recurrent neural networks 1556 (RNNs) are a natural choice for sequence model- ing tasks. Recently, RNNs …

Content truncated to 500 words