You Only Need Attention to Traverse Trees

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 316–322 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 316 You Only Need Attention to Traverse Trees Mahtab Ahmed, Muhammad Rifayat Samee and Robert E. Mercer Department of Computer Science, University of Western Ontario {mahme255, msamee, rmercer}@uwo.ca Abstract In recent NLP research, a topic of interest is universal sentence encoding, sentence rep- resentations that can be used in any super- vised task. At the word sequence level, fully attention-based models suffer from two prob- lems: a quadratic increase in memory con- sumption with respect to the sentence length and an inability to capture and use syntactic information. Recursive neural nets can extract very good syntactic information by travers- ing a tree structure. To this end, we pro- pose Tree Transformer, a model that captures phrase level syntax for constituency trees as well as word-level dependencies for depen- dency trees by doing recursive traversal only with attention. Evaluation of this model on four tasks gets noteworthy results compared to the standard transformer and LSTM-based models as well as tree-structured LSTMs. Ab- lation studies to find whether positional infor- mation is inherently encoded in the trees and which type of attention is suitable for doing the recursive traversal are provided. 1 Introduction Following the breakthrough in NLP research with word embeddings by Mikolov et al. (2013), recent research has focused on sentence representations. Having good sentence representations can help ac- complish many NLP tasks because we eventually deal with sentences, e.g., question answering, sen- timent analysis, semantic similarity, and natural language inference. Most of the existing task specific sequential sentence encoders are based on recurrent neural nets such as LSTMs or GRUs (Conneau et al., 2017; Lin et al., 2017; Liu et al., 2016). All of these works follow a common paradigm: use an LSTM/GRU over the word sequence, extract con- textual features at each time step, and apply some kind of pooling on top of that. However, a few works adopt some different methods. Kiros et al. (2015) propose a skip-gram-like objective func- tion at the sentence level to obtain the sentence embeddings. Logeswaran and Lee (2018) refor- mulate the task of predicting the next sentence given the current one into a classification problem where instead of a decoder they use a classifier to predict the next sentence from a set of candidates. The attention mechanism adopted by most of the RNN based models require access to the hid- den states at every time step (Yang et al., 2016; Kumar et al., 2016). These models are ineffi- cient and at the same time very hard to paral- lelize. To overcome this, Parikh et al. (2016) pro- pose a fully attention-based neural network which can adequately model the word dependencies and at the same time is parallelizable. Vaswani et al. (2017) adopt the multi-head version in both the encoder and decoder of their Transformer model along with positional encoding. Ahmed et al. (2017) propose …

Content truncated to 500 words