Do I have the Knowledge to Answer? Investigating Answerability of Knowledge Base Questions

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 10341–10357 July 9-14, 2023 ©2023 Association for Computational Linguistics Do I have the Knowledge to Answer? Investigating Answerability of Knowledge Base Questions Mayur Patidar†, Prayushi Faldu‡, Avinash Singh†, Lovekesh Vig†, Indrajit Bhattacharya†, Mausam‡ †TCS Research, ‡Indian Institute of Technology, Delhi {patidar.mayur, singh.avinash9, lovekesh.vig, b.indrajit} @tcs.com prayushifaldu123@gmail.com, mausam@cse.iitd.ac.in Abstract When answering natural language questions over knowledge bases, missing facts, incom- plete schema and limited scope naturally lead to many questions being unanswerable. While answerability has been explored in other QA settings, it has not been studied for QA over knowledge bases (KBQA). We create GrailQA- bility, a new benchmark KBQA dataset with unanswerability, by first identifying various forms of KB incompleteness that make ques- tions unanswerable, and then systematically adapting GrailQA (a popular KBQA dataset with only answerable questions). Experiment- ing with three state-of-the-art KBQA models, we find that all three models suffer a drop in performance even after suitable adaptation for unanswerable questions. In addition, these of- ten detect unanswerability for wrong reasons and find specific forms of unanswerability par- ticularly difficult to handle. This underscores the need for further research in making KBQA systems robust to unanswerability. 1 Introduction The problem of natural language question answer- ing over knowledge bases (KBQA) has received a lot of interest in recent years (Saxena et al., 2020; Zhang et al., 2022; Mitra et al., 2022; Wang et al., 2022; Das et al., 2022; Cao et al., 2022c; Ye et al., 2022; Chen et al., 2021; Das et al., 2021). An important aspect of this task for real-world deploy- ment is detecting answerability of questions. This problem arises for KBs due to various reasons, in- cluding schema-level and data-level incomplete- ness of KBs (Min et al., 2013), limited KB scope, questions with false premises, etc. In such cases, a robust and trustworthy model should detect and report that a question is unanswerable, instead of outputting some incorrect answer. Answerability is well studied for QA over un- structured contexts (Rajpurkar et al., 2018; Choi et al., 2018; Reddy et al., 2019; Sulem et al., 2022; Raina and Gales, 2022). However, there is no exist- ing work on answerability for KBQA. Benchmark KBQA datasets (Gu et al., 2021; Yih et al., 2016a; Talmor and Berant, 2018; Cao et al., 2022a) contain only answerable questions. We first identify how different categories of KB incompleteness (schema and data incompleteness) affect answerability of questions. Then, using GrailQA (Gu et al., 2021), one of the largest KBQA benchmark dataset, we create a new benchmark for KBQA with unanswerable questions, which we call GrailQAbility, by deleting various elements from the KB to simulate scope and fact coverage limita- tions. This involves addressing a host of challenges, arising due to different ways in which KB element deletion affects answerability of questions, depen- dence between deletion of different types of KB elements, the shared nature of KB elements across questions, and more. We also define …

Content truncated to 500 words