DAG-Structured Long Short-Term Memory for Semantic Compositionality

Proceedings of NAACL-HLT 2016, pages 917–926, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics DAG-Structured Long Short-Term Memory for Semantic Compositionality Xiaodan Zhu National Research Council Canada 1200 Montreal Road, M50 Ottawa, ON K1A 0R6, Canada zhu2048@gmail.com Parinaz Sobhani EECS, University of Ottawa 800 King Edward Avenue Ottawa, ON K1N 6N5, Canada psobh090@uottawa.ca Hongyu Guo National Research Council Canada 1200 Montreal Road, M50 Ottawa, ON K1A 0R6, Canada hongyu.guo@nrc-cnrc.gc.ca Abstract Recurrent neural networks, particularly long short-term memory (LSTM), have recently shown to be very effective in a wide range of sequence modeling problems, core to which is effective learning of distributed representation for subsequences as well as the sequences they form. An assumption in almost all the previ- ous models, however, posits that the learned representation (e.g., a distributed representa- tion for a sentence), is fully compositional from the atomic components (e.g., representa- tions for words), while non-compositionality is a basic phenomenon in human languages. In this paper, we relieve the assumption by extending the chain-structured LSTM to di- rected acyclic graphs (DAGs), with the aim to endow linear-chain LSTMs with the capa- bility of considering compositionality together with non-compositionality in the same seman- tic composition framework. From a more general viewpoint, the proposed models in- corporate additional prior knowledge into re- current neural networks, which is interesting to us, considering most NLP tasks have rela- tively small training data and appropriate prior knowledge could be beneficial to help cover missing semantics. Our experiments on sen- timent composition demonstrate that the pro- posed models achieve the state-of-the-art per- formance, outperforming models that lack this ability. 1 Introduction Recurrent neural networks, particularly long short- term memory (LSTM), have recently shown to be very effective in a wide range of sequence modeling problems, including speech recognition (Graves et al., 2013), automatic machine translation (Sutskever et al., 2014; Cho et al., 2014), and image-to-text con- version (Vinyals et al., 2014), among many others. The specific memory copying and gating configura- tions in LSTM’s memory blocks render an effective mechanism in capturing both short and distant inter- plays in an input sequence. In modeling sequences, core to many problems is to learn effective distributed representations for sub- sequences and the sequences they form. A strong as- sumption in most previous models, however, posits that the learned representation (e.g., a distributed representation for a sentence) is fully compositional from the atomic components (e.g., representations for words), while non-compositionality is a basic phenomenon in human languages and other modal- ities, which does not only include rather rigid cases such as idiomatic expressions (e.g., kick the bucket) but also soft cases that are harder to make a binary judgment. A framework with the capability to consider both compositionality and non-compositionality in se- mantic composition are of theoretic interest. From a more pragmatical viewpoint, if one is able to holisti- cally obtain the representations for a sequence (e.g., for the bigram must try in a customer-review corpus for sentiment analysis), it would be desirable that …

Content truncated to 500 words