Modelling Sentence Pairs with Tree-structured Attentive Encoder

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2912–2922, Osaka, Japan, December 11-17 2016. Modelling Sentence Pairs with Tree-structured Attentive Encoder Yao Zhou, Cong Liu∗ and Yan Pan School of Data and Computer Science, Sun Yat-sen University yoosan.zhou@gmail.com {liucong3, panyan5}@mail.sysu.edu.cn Abstract We describe an attentive encoder that combines tree-structured recursive neural networks and se- quential recurrent neural networks for modelling sentence pairs. Since existing attentive models exert attention on the sequential structure, we propose a way to incorporate attention into the tree topology. Specially, given a pair of sentences, our attentive encoder uses the representation of one sentence, which generated via an RNN, to guide the structural encoding of the other sentence on the dependency parse tree. We evaluate the proposed attentive encoder on three tasks: semantic similarity, paraphrase identification and true-false question selection. Experimental results show that our encoder outperforms all baselines and achieves state-of-the-art results on two tasks. 1 Introduction Modelling a sentence pair is to score two pieces of sentences in terms of their semantic relationship. The applications include measuring the semantic relatedness of two sentences (Marelli et al., 2014), recognizing the textual entailment (Bowman et al., 2015) between the premise and hypothesis sentences, paraphrase identification (He et al., 2015), answer selection and query ranking (Yin et al., 2015) etc. The approach of modelling a sentence pair based on neural networks usually consist of two steps. First, a sentence encoder transforms each sentence into a vector representation. Second, a classifier receives two sentence representations as features to make the classification. The sentence encoder can be regarded as a semantic compositional function which maps a sequence of word vectors to a sentence vector. This compositional function takes a range of different forms, including (but not limited to) sequential recurrent neural networks (Seq-RNNs) (Mikolov, 2012), tree-structured recursive neural networks (Tree- RNNs) (Socher et al., 2014; Tai et al., 2015) and convolutional neural networks (CNNs) (Kim, 2014). We introduce an approach that combines recursive neural networks and recurrent neural networks with the attention mechanism, which has been widely used in the sequence to sequence learning (seq2seq) framework whose applications ranges from machine translation (Bahdanau et al., 2015; Luong et al., 2015), text summarization (Rush et al., 2015) to natural language conversation (Shang et al., 2015) and other NLP tasks such as question answering (Sukhbaatar et al., 2015; Hermann et al., 2015), classifica- tion (Rockt¨aschel et al., 2016; Shimaoka et al., 2016). In the machine translation, the attention mecha- nism is used to learn the alignments between source words and target words in the decoding phase. More generally, we consider that the motivation of attention mechanism is to allow the model to attend over a set of elements with the intention of attaching different emphases to each element. We argue that the attention mechanism used in a tree-structured model is different from a sequential model. Our idea is inspired by Rockt¨aschel et al. (2016) and Hermann et al. (2015). In this paper, we utilise the …

Content truncated to 500 words