Recursive Tree-Structured Self-Attention for Answer Sentence Selection

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4651–4661 August 1–6, 2021. ©2021 Association for Computational Linguistics 4651 Recursive Tree-Structured Self-Attention for Answer Sentence Selection Khalil Mrini, Emilia Farcas, and Ndapa Nakashole University of California, San Diego, La Jolla, CA 92093 {khalil, efarcas, nnakashole}@ucsd.edu Abstract Syntactic structure is an important compo- nent of natural language text. Recent top- performing models in Answer Sentence Selec- tion (AS2) use self-attention and transfer learn- ing, but not syntactic structure. Tree struc- tures have shown strong performance in tasks with sentence pair input like semantic related- ness. We investigate whether tree structures can boost performance in AS2. We introduce the Tree Aggregation Transformer: a novel recursive, tree-structured self-attention model for AS2. The recursive nature of our model is able to represent all levels of syntactic parse trees with only one additional self-attention layer. Without transfer learning, we establish a new state of the art on the popular TrecQA and WikiQA benchmark datasets. Addition- ally, we evaluate our method on four Com- munity Question Answering datasets, and find that tree-structured representations have limi- tations with noisy user-generated text. We con- duct probing experiments to evaluate how our models leverage tree structures across datasets. Our findings show that the ability of tree- structured models to successfully absorb syn- tactic information is strongly correlated with a higher performance in AS2. 1 Introduction Motivation. Natural language text is character- ized by structure. For instance, syntactic parse trees decompose a sentence into syntactic groups, which in turn are decomposed recursively until we get to single-word spans. Therefore, syntactic parse trees have a varying number of levels that can be accurately represented by recursive model architec- tures. Tree-structured LSTM networks (Tai et al., 2015) are the recursive extension of LSTM net- works (Hochreiter and Schmidhuber, 1997), and allow for syntactic trees to be represented hierarchi- cally. Tree-LSTMs and bidirectional Tree-LSTMs <s> Marrakesh </s> <s> in Marrakesh </s> <s> died in Marrakesh </s> <s> Averroes </s> <s> Averroes died in Marrakesh. </s> Self-Attention Self-Attention Self-Attention Self-Attention Self-Attention Sentence Embedding Figure 1: Embedding a sentence with our proposed recursive tree-structured self-attention using the corre- sponding constituency parse tree. There is only one set of parameters for the recursive self-attention. (Teng and Zhang, 2017) do not represent sequence position information, whereas the hybrid neural inference networks (Chen et al., 2017a) represent sequence position information separately from tree- structured hierarchical information. Tree-structured models have been applied to the tasks of natural language inference (Chen et al., 2017a), sentence pair similarity (Tai et al., 2015), dependency parsing (Kiperwasser and Goldberg, 2016), and text embeddings (Mrini et al., 2019). In this paper, we consider the problem of Answer Sen- tence Selection (AS2), where the goal is to predict for a question-sentence pair whether the sentence contains an answer to the question. Given that tree- structured models have performed strongly on a task that takes a sentence pair as input – sentence …

Content truncated to 500 words