Compositional Distributional Semantics with Long Short Term Memory

Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics (*SEM 2015), pages 10–19, Denver, Colorado, June 4–5, 2015. Compositional Distributional Semantics with Long Short Term Memory Phong Le and Willem Zuidema Institute for Logic, Language and Computation University of Amsterdam, the Netherlands {p.le,zuidema}@uva.nl Abstract We are proposing an extension of the recur- sive neural network that makes use of a vari- ant of the long short-term memory architec- ture. The extension allows information low in parse trees to be stored in a memory reg- ister (the ‘memory cell’) and used much later higher up in the parse tree. This provides a so- lution to the vanishing gradient problem and allows the network to capture long range de- pendencies. Experimental results show that our composition outperformed the traditional neural-network composition on the Stanford Sentiment Treebank. 1 Introduction Moving from lexical to compositional semantics in vector-based semantics requires answers to two dif- ficult questions: (i) what is the nature of the com- position functions (given that the lambda calculus for variable binding is no longer applicable), and (ii) how do we learn the parameters of those functions (if they have any) from data? A number of classes of functions have been proposed in answer to the first question, including simple linear functions like vec- tor addition (Mitchell and Lapata, 2009), non-linear functions like those defined by multi-layer neural networks (Socher et al., 2010), and vector matrix multiplication and tensor linear mapping (Baroni et al., 2013). The matrix and tensor-based functions have the advantage of allowing a relatively straight- forward comparison with formal semantics, but the fact that multi-layer neural networks with non-linear activation functions like sigmoid can approximate any continuous function (Cybenko, 1989) already make them an attractive choice. In trying to answer the second question, the ad- vantages of approaches based on neural network ar- chitectures, such as the recursive neural network (RNN) model (Socher et al., 2013b) and the con- volutional neural network model (Kalchbrenner et al., 2014), are even clearer. Models in this paradigm can take advantage of general learning procedures based on back-propagation, and with the rise of ‘deep learning’, of a variety of efficient algorithms and tricks to further improve training. Since the first success of the RNN model (Socher et al., 2011b) in constituent parsing, two classes of extensions have been proposed. One class is to en- hance its compositionality by using tensor product (Socher et al., 2013b) or concatenating RNNs hor- izontally to make a deeper net (Irsoy and Cardie, 2014). The other is to extend its topology in order to fulfill a wider range of tasks, like Le and Zuidema (2014a) for dependency parsing and Paulus et al. (2014) for context-dependence sentiment analysis. Our proposal in this paper is an extension of the RNN model to improve compositionality. Our mo- tivation is that, like training recurrent neural net- works, training RNNs on deep trees can suffer from the vanishing gradient problem (Hochreiter et al., 2001), i.e., that errors propagated back to the …

Content truncated to 500 words