HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1026–1036 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1026 HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, William Wang University of California, Santa Barbara, CA, USA {wenhuchen, hwzha, xwhan}@cs.ucsb.edu, {zhiyuchen, hongwang600, william}@cs.ucsb.edu Abstract Existing question answering datasets focus on dealing with homogeneous information, based either only on text or KB/Table information alone. However, as human knowledge is dis- tributed over heterogeneous forms, using ho- mogeneous information alone might lead to severe coverage problems. To fill in the gap, we present HybridQA1, a new large-scale question-answering dataset that requires rea- soning on heterogeneous information. Each question is aligned with a Wikipedia table and multiple free-form corpora linked with the en- tities in the table. The questions are designed to aggregate both tabular information and text information, i.e., lack of either form would ren- der the question unanswerable. We test with three different models: 1) a table-only model. 2) text-only model. 3) a hybrid model that combines heterogeneous information to find the answer. The experimental results show that the EM scores obtained by two baselines are below 20%, while the hybrid model can achieve an EM over 40%. This gap suggests the necessity to aggregate heterogeneous in- formation in HybridQA. However, the hybrid model’s score is still far behind human perfor- mance. Hence, HybridQA can serve as a chal- lenging benchmark to study question answer- ing with heterogeneous information. 1 Introduction Question answering systems aim to answer any form of question of our interests, with evidence provided by either free-form text like Wikipedia passages (Rajpurkar et al., 2016; Chen et al., 2017; Yang et al., 2018) or structured data like Free- base/WikiData (Berant et al., 2013; Kwiatkowski et al., 2013; Yih et al., 2015; Weston et al., 2015) and WikiTables (Pasupat and Liang, 2015). Both 1https://github.com/wenhuchen/HybridQA forms have their advantages, the free-form cor- pus has in general better coverage while structured data has better compositionality to handle complex multi-hop questions. Due to the advantages of dif- ferent representation forms, people like to combine them in real world applications. Therefore, it is sometime not ideal to assume the question has an- swer in a passage. This paper aims to simulate a more realistic setting where the evidences are dis- tributed into heterogeneous data, and the model requires to aggregate information from different forms for answering a question. There has been some pioneering work on building hybrid QA sys- tems (Sun et al., 2019, 2018; Xiong et al., 2019). These methods adopts KB-only datasets (Berant et al., 2013; Yih et al., 2015; Talmor and Berant, 2018) to simulate a hybrid setting by randomly masking KB triples and replace them with text corpus. Experimental results have proved decent improvement, which shed lights on the potential of hybrid question answering systems to integrate heterogeneous information. Though there already exist numerous valuable questions answering datasets as listed in Table 1, …

Content truncated to 500 words