Recovering Question Answering Errors via Query Revision

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 903–909 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Recovering Question Answering Errors via Query Revision Semih Yavuz and Izzeddin Gur and Yu Su and Xifeng Yan Department of Computer Science, University of California, Santa Barbara {syavuz,izzeddingur,ysu,xyan}@cs.ucsb.edu Abstract The existing factoid QA systems often lack a post-inspection component that can help models recover from their own mis- takes. In this work, we propose to cross- check the corresponding KB relations be- hind the predicted answers and identify potential inconsistencies. Instead of devel- oping a new model that accepts evidences collected from these relations, we choose to plug them back to the original questions directly and check if the revised question makes sense or not. A bidirectional LSTM is applied to encode revised questions. We develop a scoring mechanism over the re- vised question encodings to refine the pre- dictions of a base QA system. This ap- proach can improve the F1 score of STAGG (Yih et al., 2015), one of the leading QA systems, from 52.5% to 53.9% on WE- BQUESTIONS data. 1 Introduction With the recent advances in building large scale knowledge bases (KB) like Freebase (Bollacker et al., 2008), DBpedia (Auer et al., 2007), and YAGO (Suchanek et al., 2007) that contain the world’s factual information, KB-based question answering receives attention of research efforts in this area. Traditional semantic parsing is one of the most promising approaches that tackles this problem by mapping questions onto logical forms using logical languages CCG (Kwiatkowski et al., 2013; Reddy et al., 2014; Choi et al., 2015; Reddy et al., 2016), DCS (Berant et al., 2013; Berant and Liang, 2014, 2015), or directly query graphs (Yih et al., 2015) with predicates closely related to KB schema. Recently, neural network based models have been applied to question answering (Bordes Figure 1: Sketch of our approach. Elements in solid round rectangles are KB relation labels. Relation on the left is cor- rect, but the base QA system predicts the one on the right. Dotted rectangles represent revised questions with relation labels plugged in. The left revised question looks semanti- cally closer to the original question and itself is more consis- tent. Hence, it shall be ranked higher than the right one. et al., 2015; Yih et al., 2015; Xu et al., 2016a,b). While these approaches yielded successful re- sults, they often lack a post-inspection component that can help models recover from their own mis- takes. Table 1 shows the potential improvement we can achieve if such a component exists. Can we leverage textual evidences related to the pre- dicted answers to recover from a prediction error? In this work, we show it is possible. Our strategy is to cross-check the correspond- ing KB relations behind the predicted answers and identify potential inconsistencies. As an interme- diate step, we define question revision as a tai- lored transformation of the original question using textual evidences collected from these relations in a …

Content truncated to 500 words