A Fast Unified Model for Parsing and Sentence Understanding

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1466–1477, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics A Fast Unified Model for Parsing and Sentence Understanding Samuel R. Bowman1,2,3,∗ sbowman@stanford.edu Jon Gauthier2,3,4,∗ jgauthie@stanford.edu Abhinav Rastogi3,5 arastogi@stanford.edu Raghav Gupta2,3,6 rgupta93@stanford.edu Christopher D. Manning1,2,3,6 manning@stanford.edu Christopher Potts1,6 cgpotts@stanford.edu 1Stanford Linguistics 2Stanford NLP Group 3Stanford AI Lab 4Stanford Symbolic Systems 5Stanford Electrical Engineering 6Stanford Computer Science Abstract Tree-structured neural networks exploit valuable syntactic parse information as they interpret the meanings of sentences. However, they suffer from two key techni- cal problems that make them slow and un- wieldy for large-scale NLP tasks: they usu- ally operate on parsed sentences and they do not directly support batched computa- tion. We address these issues by introduc- ing the Stack-augmented Parser-Interpreter Neural Network (SPINN), which combines parsing and interpretation within a single tree-sequence hybrid model by integrating tree-structured sentence interpretation into the linear sequential structure of a shift- reduce parser. Our model supports batched computation for a speedup of up to 25× over other tree-structured models, and its integrated parser can operate on unparsed data with little loss in accuracy. We evalu- ate it on the Stanford NLI entailment task and show that it significantly outperforms other sentence-encoding models. 1 Introduction A wide range of current models in NLP are built around a neural network component that produces vector representations of sentence meaning (e.g., Sutskever et al., 2014; Tai et al., 2015). This com- ponent, the sentence encoder, is generally formu- lated as a learned parametric function from a se- quence of word vectors to a sentence vector, and this function can take a range of different forms. Common sentence encoders include sequence- based recurrent neural network models (RNNs, see Figure 1a) with Long Short-Term Memory (LSTM, ∗The first two authors contributed equally. the old cat ate ... the cat sat down ... (a) A conventional sequence-based RNN for two sentences. ... the old cat ate ate the old cat old cat cat old the ... the cat sat down sat down down sat the cat cat the (b) A conventional TreeRNN for two sentences. Figure 1: An illustration of two standard designs for sentence encoders. The TreeRNN, unlike the sequence-based RNN, requires a substantially dif- ferent connection structure for each sentence, mak- ing batched computation impractical. Hochreiter and Schmidhuber, 1997), which ac- cumulate information over the sentence sequen- tially; convolutional neural networks (Kalchbren- ner et al., 2014; Zhang et al., 2015), which accu- mulate information using filters over short local se- quences of words or characters; and tree-structured recursive neural networks (TreeRNNs, Goller and Küchler, 1996; Socher et al., 2011a, see Figure 1b), which propagate information up a binary parse tree. Of these, the TreeRNN appears to be the prin- cipled choice, since meaning in natural language sentences is known to be constructed recursively according to a tree structure (Dowty, 2007, i.a.). TreeRNNs have shown promise (Tai et al., 2015; Li et al., 2015; Bowman et al., 2015b), …

Content truncated to 500 words