TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8413–8426 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 8413 TABERT: Pretraining for Joint Understanding of Textual and Tabular Data Pengcheng Yin∗ Graham Neubig Carnegie Mellon University {pcyin,gneubig}@cs.cmu.edu Wen-tau Yih Sebastian Riedel Facebook AI Research {scottyih,sriedel}@fb.com Abstract Recent years have witnessed the burgeoning of pretrained language models (LMs) for text- based natural language (NL) understanding tasks. Such models are typically trained on free-form NL text, hence may not be suit- able for tasks like semantic parsing over struc- tured data, which require reasoning over both free-form NL questions and structured tabular data (e.g., database tables). In this paper we present TABERT, a pretrained LM that jointly learns representations for NL sentences and (semi-)structured tables. TABERT is trained on a large corpus of 26 million tables and their English contexts. In experiments, neural se- mantic parsers using TABERT as feature rep- resentation layers achieve new best results on the challenging weakly-supervised semantic parsing benchmark WIKITABLEQUESTIONS, while performing competitively on the text-to- SQL dataset SPIDER.1 1 Introduction Recent years have witnessed a rapid advance in the ability to understand and answer questions about free-form natural language (NL) text (Rajpurkar et al., 2016), largely due to large-scale, pretrained language models (LMs) like BERT (Devlin et al., 2019). These models allow us to capture the syntax and semantics of text via representations learned in an unsupervised manner, before fine-tuning the model to downstream tasks (Melamud et al., 2016; McCann et al., 2017; Peters et al., 2018; Liu et al., 2019b; Yang et al., 2019; Goldberg, 2019). It is also relatively easy to apply such pretrained LMs to comprehension tasks that are modeled as text span selection problems, where the boundary of an answer span can be predicted using a simple classifier on top of the LM (Joshi et al., 2019). ∗Work done while at Facebook AI Research. 1Available at github.com/facebookresearch/TaBERT However, it is less clear how one could pretrain and fine-tune such models for other QA tasks that involve joint reasoning over both free-form NL text and structured data. One example task is seman- tic parsing for access to databases (DBs) (Zelle and Mooney, 1996; Berant et al., 2013; Yih et al., 2015), the task of transducing an NL utterance (e.g., “Which country has the largest GDP?”) into a struc- tured query over DB tables (e.g., SQL querying a database of economics). A key challenge in this scenario is understanding the structured schema of DB tables (e.g., the name, data type, and stored val- ues of columns), and more importantly, the align- ment between the input text and the schema (e.g., the token “GDP” refers to the Gross Domestic Product column), which is essential for inferring the correct DB query (Berant and Liang, 2014). Neural semantic parsers tailored to this task therefore attempt to learn joint representations of NL utterances and the (semi-)structured schema of DB tables (e.g., representations of its columns or cell values, as in Krishnamurthy et …

Content truncated to 500 words