OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 932 - 942 July 10-15, 2022 ©2022 Association for Computational Linguistics OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering Zhengbao Jiang1∗, Yi Mao2, Pengcheng He2, Graham Neubig1, Weizhu Chen2 1Language Technologies Institute, Carnegie Mellon University {zhengbaj,gneubig}@cs.cmu.edu2Microsoft Azure AI {maoyi,penhe,wzchen}@microsoft.com Abstract The information in tables can be an important complement to text, making table-based ques- tion answering (QA) systems of great value. The intrinsic complexity of handling tables of- ten adds an extra burden to both model design and data annotation. In this paper, we aim to develop a simple table-based QA model with minimal annotation effort. Motivated by the fact that table-based QA requires both align- ment between questions and tables and the ability to perform complicated reasoning over multiple table elements, we propose an om- nivorous pretraining approach that consumes both natural and synthetic data to endow mod- els with these respective abilities. Specifically, given freely available tables, we leverage re- trieval to pair them with relevant natural sen- tences for mask-based pretraining, and syn- thesize NL questions by converting SQL sam- pled from tables for pretraining with a QA loss. We perform extensive experiments in both few-shot and full settings, and the re- sults clearly demonstrate the superiority of our model OmniTab, with the best multitasking ap- proach achieving an absolute gain of 16.2% and 2.7% in 128-shot and full settings respec- tively, also establishing a new state-of-the-art on WikiTableQuestions. Detailed ablations and analyses reveal different characteristics of natural and synthetic data, shedding light on future directions in omnivorous pretraining.1 1 Introduction Humans are voracious information omnivores, con- suming information in a number of forms. Un- structured text is the form covered in most work in NLP, but another form widely used on the web, academic papers, and reports is the table, where elements are organized in rows and columns and presented in a structured and succinct way. Be- cause of this, systems to aid information access ∗Work was done when interning at Microsoft Azure AI. 1Code, pretraining data, and pretrained models are avail- able at https://github.com/jzbjyb/OmniTab. over tables, such as table-based question answering (QA) (Pasupat and Liang, 2015; Iyyer et al., 2017; Zhong et al., 2017), hold significant promise. How- ever, they also require understanding of the table structure and sophisticated reasoning across multi- ple elements to get the final answer. This intrinsic complexity not only often requires special-purpose model designs such as pipeline systems that gener- ate structured queries as intermediate steps (Liang et al., 2018; Wang et al., 2019; Yin et al., 2020; Yu et al., 2021), but also adds an extra burden to the process of data annotation (Pasupat and Liang, 2015; Shi et al., 2020). Given the challenges above, we ask the ques- tion: “can we create a simple model that is able to answer complex table-based questions with mini- mal annotation effort?” Both modeling simplicity and limited assumptions …

Content truncated to 500 words