On Generating Characteristic-rich Question Sets for QA Evaluation

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 562–572, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics On Generating Characteristic-rich Question Sets for QA Evaluation Yu Su1, Huan Sun2, Brian Sadler3, Mudhakar Srivatsa4 Izzeddin G¨ur1, Zenghui Yan1 and Xifeng Yan1 1University of California, Santa Barbara, Department of Computer Science 2The Ohio State University, Department of Computer Science and Engineering 3U.S. Army Research Lab, 4IBM Research {ysu,izzeddingur,zyan,xyan}@cs.ucsb.edu, sun.397@osu.edu brian.m.sadler6.civ@mail.mil, msrivats@us.ibm.com Abstract We present a semi-automated framework for constructing factoid question answering (QA) datasets, where an array of question character- istics are formalized, including structure com- plexity, function, commonness, answer cardi- nality, and paraphrasing. Instead of collecting questions and manually characterizing them, we employ a reverse procedure, first generat- ing a kind of graph-structured logical forms from a knowledge base, and then converting them into questions. Our work is the first to generate questions with explicitly specified characteristics for QA evaluation. We con- struct a new QA dataset with over 5,000 log- ical form-question pairs, associated with an- swers from the knowledge base, and show that datasets constructed in this way enable fine- grained analyses of QA systems. The dataset can be found in https://github.com/ ysu1989/GraphQuestions. 1 Introduction Factoid question answering (QA) has gained great attention recently, owing to the fast growth of large knowledge bases (KBs) such as DBpedia (Lehmann et al., 2014) and Freebase (Bollacker et al., 2008), which avail QA systems of comprehensive and pre- cise knowledge of encyclopedic scope (Yahya et al., 2012; Berant et al., 2013; Cai and Yates, 2013; Kwiatkowski et al., 2013; Berant and Liang, 2014; Fader et al., 2014; Reddy et al., 2014; Bao et al., 2014; Zou et al., 2014; Yao and Van Durme, 2014; Yih et al., 2015; Sun et al., 2015; Dong et al., 2015; Yao, 2015; Berant and Liang, 2015). With the blos- soming of QA systems, evaluation is becoming an increasingly important problem. QA datasets, con- sisting of a set of questions with ground-truth an- swers, are critical for both comparing existing sys- tems and gaining insights to develop new systems. Questions have rich characteristics, constituting dimensions along which question difficulty varies. Some questions are difficult due to their com- plex semantic structure (“Who was the coach when Michael Jordan stopped playing for the Chicago Bulls?”), while some others may be difficult because they require a precise quantitative analysis over the answer space (“What is the best-selling smartphone in 2015?”). Many other characteristics shall be con- sidered too, e.g., what topic a question is about (questions about common topics may be easier to answer) and how many answers there are (it is harder to achieve a high recall in case of multiple answers). Worse still, due to the flexibility of natural language, different people often describe the same question in different ways, i.e., paraphrasing. It is important for a QA system to be robust to paraphrasing. A QA dataset explicitly specifying such ques- tion characteristics allows for fine-grained inspec- tion of system …

Content truncated to 500 words