Topic Transferable Table Question Answering

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4159–4172 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4159 Topic Transferable Table Question Answering Saneem A. Chemmengath1∗ Vishwajeet Kumar1∗ Samarth Bharadwaj1 Jaydeep Sen1 Mustafa Canim1, Soumen Chakrabarti2 Alfio Gliozzo1 Karthik Sankaranarayanan1, 1IBM Research 2Indian Institute of Technology, Bombay saneem.cg@in.ibm.com, vishk024@in.ibm.com, samarth.b@in.ibm.com, jaydesen@in.ibm.com, mustafa@us.ibm.com, soumen.chakrabarti@gmail.com gliozzo@us.ibm.com, kartsank@in.ibm.com Abstract Weakly-supervised table question-answering (TableQA) models have achieved state-of-art performance by using pre-trained BERT trans- former to jointly encoding a question and a ta- ble to produce structured query for the ques- tion. However, in practical settings TableQA systems are deployed over table corpora hav- ing topic and word distributions quite dis- tinct from BERT’s pretraining corpus. In this work we simulate the practical topic shift scenario by designing novel challenge bench- marks WikiSQL-TS and WikiTQ-TS1, con- sisting of train-dev-test splits in five distinct topic groups, based on the popular WikiSQL and WikiTableQuestions datasets. We empir- ically show that, despite pre-training on large open-domain text, performance of models de- grades significantly when they are evaluated on unseen topics. In response, we propose T3QA (Topic Transferable Table Question An- swering) a pragmatic adaptation framework for TableQA comprising of: (1) topic-specific vocabulary injection into BERT, (2) a novel text-to-text transformer generator (such as T5, GPT2) based natural language question gener- ation pipeline focused on generating topic spe- cific training data, and (3) a logical form re- ranker. We show that T3QA provides a reason- ably good baseline for our topic shift bench- marks. We believe our topic split benchmarks will lead to robust TableQA solutions that are better suited for practical deployment. 1 Introduction Documents, particularly in enterprise settings, of- ten contain valuable tabular information (e.g., finan- cial, sales/marketing, HR). Natural language ques- tion answering systems over a table (or TableQA) have an additional complexity of understanding the tabular structure including row/column head- ∗Equal contribution by first two authors. 1The source code and new dataset splits are available at https://github.com/IBM/T3QA Party Candidate Votes Conservatives Andrew Turner 32,717 Liberal Democrats Anthony Rowlands 19,739 Labour Mark Chiverton 11,484 UK Independence Michael Tarrant 2,352 Independent Edward Corby 551 Question: Who ran in the election for labour party? Answer : Mark Chiverton Figure 1: Topic-sensitive representations are important to infer that, in the context of the topic politics, the query span “ran in the election” should be linked to the “Candidate” column in the table. ers compared to the more widely-studied passage- based reading comprehension (RC) problem. Fur- ther, TableQA may involve complex questions with multi-cell or aggregate answers. Most of the TableQA systems use semantic pars- ing approaches that utilizes language encoders to produce an intermediate logical form from the nat- ural language question which is executed that over the tabular data to get the answer. While some sys- tems (Zhong et al., 2017) were fully supervised, needing pairs of questions and logical forms as training data, more recent systems (Pasupat and Liang, 2015; Krishnamurthy et al., 2017; Dasigi et al., 2019) rely only on …

Content truncated to 500 words