If You Can’t Beat Them Join Them: Handcrafted Features Complement Neural Nets for Non-Factoid Answer Reranking

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 121–131, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics If You Can’t Beat Them Join Them: Handcrafted Features Complement Neural Nets for Non-Factoid Answer Reranking Dasha Bogdanova, Jennifer Foster, Daria Dzendzik and Qun Liu ADAPT Centre School of Computing, Dublin City University Dublin, Ireland firstname.lastname@adaptcentre.ie Abstract We show that a neural approach to the task of non-factoid answer reranking can ben- efit from the inclusion of tried-and-tested handcrafted features. We present a novel neural network architecture based on a combination of recurrent neural networks that are used to encode questions and an- swers, and a multilayer perceptron. We show how this approach can be combined with additional features, in particular, the discourse features presented by Jansen et al. (2014). Our neural approach achieves state-of-the-art performance on a public dataset from Yahoo! Answers and its per- formance is further improved by incorpo- rating the discourse features. Additionally, we present a new dataset of Ask Ubuntu questions where the hybrid approach also achieves good results. 1 Introduction The task of Question Answering (QA) is arguably one of the oldest tasks in Natural Language Pro- cessing (NLP), attracting high levels of interest from both industry and academia. The QA track at the Text Retrieval Evaluation Conference (TREC) was introduced in 1999 and since then has encour- aged many research studies by providing a plat- form for evaluation and making labeled datasets available. However, most research has focused on factoid questions, e.g. the TREC questions What is the name of the managing director of Apricot Computer? and What was the monetary value of the Nobel Prize in 1989? The TREC QA track organizers took care to “select questions with straightforward, obvious answers” (Voorhees and Tice, 1999) to facilitate manual assessment. In contrast, research on answering non-factoid (NF) questions, such as manner, reason, difference and opinion questions, has been rather piecemeal. This was largely due to the absence of available labeled data for the task. This is changing, however, with the growing popularity of Community Question Answering (CQA) websites, such as Quora,1 Ya- hoo! Answers2 and the Stack Exchange3 family of forums. One of the main components of a non-factoid question answering system is the answer reranking module. Given a question, it aims to rearrange the answers in order to boost the community-selected best answer to the top position. Most previous attempts to perform non-factoid answer rerank- ing on CQA data are supervised, feature-based, learning-to-rank approaches (Jansen et al., 2014; Fried et al., 2015; Sharp et al., 2015). These meth- ods represent the candidate answers as meaningful handcrafted features based on syntactic, seman- tic and discourse parses (Surdeanu et al., 2011; Jansen et al., 2014), web correlation (Surdeanu et al., 2011), and translation probabilities (Fried et al., 2015; Surdeanu et al., 2011). The result- ing feature vectors are then passed to a supervised ranking algorithm, such as SVMrank (Joachims, 2006), which ranks the …

Content truncated to 500 words