KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Base

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6101 - 6119 May 22-27, 2022 c⃝2022 Association for Computational Linguistics KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Base Shulin Cao1,2∗, Jiaxin Shi1,3∗, Liangming Pan4, Lunyiu Nie1, Yutong Xiang5, Lei Hou1,2† , Juanzi Li1,2, Bin He6, Hanwang Zhang7 1Department of Computer Science and Technology, BNRist 2KIRC, Institute for Artificial Intelligence, Tsinghua University, Beijing 100084, China 3Cloud BU, Huawei Technologies, 4National University of Singapore, 5ETH Zürich 6Noah’s Ark Lab, Huawei Technologies, 7Nanyang Technological University {caosl19@mails., houlei@,lijuanzi@}tsinghua.edu.cn shijx12@gmail.com, liangmingpan@u.nus.edu Abstract Complex question answering over knowledge base (Complex KBQA) is challenging because it requires various compositional reasoning ca- pabilities, such as multi-hop inference, attribute comparison, set operation. Existing bench- marks have some shortcomings that limit the development of Complex KBQA: 1) they only provide QA pairs without explicit reasoning processes; 2) questions are poor in diversity or scale. To this end, we introduce KQA Pro, a dataset for Complex KBQA including ~120K diverse natural language questions. We introduce a compositional and interpretable programming language KoPL to represent the reasoning process of complex questions. For each question, we provide the corresponding KoPL program and SPARQL query, so that KQA Pro serves for both KBQA and semantic parsing tasks. Experimental results show that SOTA KBQA methods cannot achieve promis- ing results on KQA Pro as on current datasets, which suggests that KQA Pro is challenging and Complex KBQA requires further research efforts. We also treat KQA Pro as a diagnos- tic dataset for testing multiple reasoning skills, conduct a thorough evaluation of existing mod- els and discuss further directions for Complex KBQA. Our codes and datasets can be obtained from https://github.com/shijx12/ KQAPro_Baselines. 1 Introduction Thanks to the recent advances in deep models, es- pecially large-scale unsupervised representation learning (Devlin et al., 2019), question answering of simple questions over knowledge base (Simple KBQA), i.e., single-relation factoid questions (Bor- des et al., 2015), begins to saturate (Petrochuk and Zettlemoyer, 2018; Wu et al., 2019; Huang ∗ indicates equal contribution † Corresponding Author et al., 2019). However, tackling complex ques- tions (Complex KBQA) is still an ongoing chal- lenge, due to the unsatisfied capability of composi- tional reasoning. As shown in Table 1, to promote the community development, several benchmarks are proposed for Complex KBQA, including LC- QuAD2.0 (Dubey et al., 2019), ComplexWebQues- tions (Talmor and Berant, 2018), MetaQA (Zhang et al., 2018), CSQA (Saha et al., 2018), CFQ (Key- sers et al., 2020), and so on. However, they suffer from the following problems: 1) Most of them only provide QA pairs without explicit reasoning processes, making it challenging for models to learn compositional reasoning. Some researchers try to learn the reasoning processes with reinforcement learning (Liang et al., 2017; Saha et al., 2019; Ansari et al., 2019) and search- ing (Guo et al., 2018). However, the prohibitively huge search space hinders both the performance and speed, especially when the question complexity increases. For …

Content truncated to 500 words