Neural Tree Indexers for Text Understanding

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 11–21, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics Neural Tree Indexers for Text Understanding Tsendsuren Munkhdalai and Hong Yu University of Massachusetts, MA, USA {tsendsuren.munkhdalai,hong.yu}@umassmed.edu Abstract Recurrent neural networks (RNNs) pro- cess input text sequentially and model the conditional transition between word to- kens. In contrast, the advantages of recur- sive networks include that they explicitly model the compositionality and the recur- sive structure of natural language. How- ever, the current recursive architecture is limited by its dependence on syntactic tree. In this paper, we introduce a robust syntactic parsing-independent tree struc- tured model, Neural Tree Indexers (NTI) that provides a middle ground between the sequential RNNs and the syntactic tree- based recursive models. NTI constructs a full n-ary tree by processing the input text with its node function in a bottom-up fash- ion. Attention mechanism can then be ap- plied to both structure and node function. We implemented and evaluated a binary- tree model of NTI, showing the model achieved the state-of-the-art performance on three different NLP tasks: natural lan- guage inference, answer sentence selec- tion, and sentence classification, outper- forming state-of-the-art recurrent and re- cursive neural networks 1. 1 Introduction Recurrent neural networks (RNNs) have been suc- cessful for modeling sequence data (Elman, 1990). RNNs equipped with gated hidden units and in- ternal short-term memories, such as long short- term memories (LSTM) (Hochreiter and Schmid- huber, 1997) have achieved a notable success in 1Code for the experiments and NTI is available at https://bitbucket.org/tsendeemts/nti several NLP tasks including named entity recog- nition (Lample et al., 2016), constituency parsing (Vinyals et al., 2015), textual entailment recogni- tion (Rockt¨aschel et al., 2016), question answer- ing (Hermann et al., 2015), and machine trans- lation (Bahdanau et al., 2015). However, most LSTM models explored so far are sequential. It encodes text sequentially from left to right or vice versa and do not naturally support compositional- ity of language. Sequential LSTM models seem to learn syntactic structure from the natural language however their generalization on unseen text is rel- atively poor comparing with models that exploit syntactic tree structure (Bowman et al., 2015b). Unlike sequential models, recursive neural net- works compose word phrases over syntactic tree structure and have shown improved performance in sentiment analysis (Socher et al., 2013). How- ever its dependence on a syntactic tree architecture limits practical NLP applications. In this study, we introduce Neural Tree Indexers (NTI), a class of tree structured models for NLP tasks. NTI takes a sequence of tokens and produces its rep- resentation by constructing a full n-ary tree in a bottom-up fashion. Each node in NTI is associated with one of the node transformation functions: leaf node mapping and non-leaf node composition functions. Unlike previous recursive models, the tree structure for NTI is relaxed, i.e., NTI does not require the input sequences to be parsed syntac- tically; and therefore it is flexible …

Content truncated to 500 words