An End-to-End Deep Framework for Answer Triggering with a Novel Group-Level Objective

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1276–1282 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics An End-to-End Deep Framework for Answer Triggering with a Novel Group-Level Objective Jie Zhao The Ohio State University zhao.1359@osu.edu Yu Su University of California, Santa Babara ysu@cs.ucsb.edu Ziyu Guan Northwest University, China ziyuguan@nwu.edu.cn Huan Sun The Ohio State University sun.397@osu.edu Abstract Given a question and a set of answer candidates, answer triggering determines whether the candidate set contains any correct answers. If yes, it then outputs a correct one. In contrast to existing pipeline methods which first consider in- dividual candidate answers separately and then make a prediction based on a thresh- old, we propose an end-to-end deep neu- ral network framework, which is trained by a novel group-level objective function that directly optimizes the answer trig- gering performance. Our objective func- tion penalizes three potential types of er- ror and allows training the framework in an end-to-end manner. Experimental re- sults on the WIKIQA benchmark show that our framework outperforms the state of the arts by a 6.6% absolute gain under F1 measure1. 1 Introduction Question Answering (QA) aims at automatically responding to natural language questions with di- rect answers (Heilman and Smith, 2010; Severyn and Moschitti, 2013; Yao et al., 2013; Berant and Liang, 2014; Yih et al., 2015; Sun et al., 2015; Miller et al., 2016; Sun et al., 2016). Most existing QA systems always output an answer for any ques- tion, no matter whether their answer candidate set contains correct answers or not (Feng et al., 2015; Severyn and Moschitti, 2015; Yang et al., 2016; Rao et al., 2016). In practice, however, this can greatly hurt user experience, especially when it is hard for users to judge answer correctness. In this paper, we study the critical yet under-addressed 1Our code is available at https://github.com/ jiez-osu/answer-triggering. Answer Triggering (Yang et al., 2015) problem: Given a question and a set of answer candidates, determine whether the candidate set contains any correct answer, and if so, select a correct answer as system output. The answer triggering problem can be logi- cally divided into two sub-problems: P1: Build an individual-level model to rank answer candi- dates so that a correct one (if it exists) gets the highest score. P2: Make a group-level binary pre- diction on the existence of correct answers within the candidate set. Previous work (Yang et al., 2015; Jurczyk et al., 2016) attack the problem via a pipeline approach: First solve P1 as a ranking task and then solve P2 by choosing an optimal threshold upon the previous step’s highest ranking score. However, the yielded answer triggering per- formance is far from satisfactory, with F1 between 32% and 36%. An alternative pipeline approach is to first solve P2 and then P1, i.e., first determine whether there’s a correct answer in the candidate set and then rank all candidates to find a correct one. However, as we will show using state-of-the- art Multiple Instance …

Content truncated to 500 words