TaPas: Weakly Supervised Table Parsing via Pre-training

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4320–4333 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4320 TAPAS: Weakly Supervised Table Parsing via Pre-training Jonathan Herzig1,2, Paweł Krzysztof Nowak1, Thomas M¨uller1, Francesco Piccinno1, Julian Martin Eisenschlos1 1Google Research 2School of Computer Science, Tel-Aviv University {jherzig,pawelnow,thomasmueller,piccinno,eisenjulian}@google.com Abstract Answering natural language questions over ta- bles is usually seen as a semantic parsing task. To alleviate the collection cost of full logical forms, one popular approach focuses on weak supervision consisting of denotations instead of logical forms. However, training seman- tic parsers from weak supervision poses diffi- culties, and in addition, the generated logical forms are only used as an intermediate step prior to retrieving the denotation. In this pa- per, we present TAPAS, an approach to ques- tion answering over tables without generating logical forms. TAPAS trains from weak super- vision, and predicts the denotation by select- ing table cells and optionally applying a cor- responding aggregation operator to such selec- tion. TAPAS extends BERT’s architecture to encode tables as input, initializes from an ef- fective joint pre-training of text segments and tables crawled from Wikipedia, and is trained end-to-end. We experiment with three differ- ent semantic parsing datasets, and find that TAPAS outperforms or rivals semantic parsing models by improving state-of-the-art accuracy on SQA from 55.1 to 67.2 and performing on par with the state-of-the-art on WIKISQL and WIKITQ, but with a simpler model architec- ture. We additionally find that transfer learn- ing, which is trivial in our setting, from WIK- ISQL to WIKITQ, yields 48.7 accuracy, 4.2 points above the state-of-the-art. 1 Introduction Question answering from semi-structured tables is usually seen as a semantic parsing task where the question is translated to a logical form that can be executed against the table to retrieve the correct denotation (Pasupat and Liang, 2015; Zhong et al., 2017; Dasigi et al., 2019; Agarwal et al., 2019). Semantic parsers rely on supervised training data that pairs natural language questions with logical forms, but such data is expensive to annotate. In recent years, many attempts aim to reduce the burden of data collection for semantic parsing, including paraphrasing (Wang et al., 2015), human in the loop (Iyer et al., 2017; Lawrence and Rie- zler, 2018) and training on examples from other domains (Herzig and Berant, 2017; Su and Yan, 2017). One prominent data collection approach focuses on weak supervision where a training ex- ample consists of a question and its denotation instead of the full logical form (Clarke et al., 2010; Liang et al., 2011; Artzi and Zettlemoyer, 2013). Although appealing, training semantic parsers from this input is often difficult due to the abundance of spurious logical forms (Berant et al., 2013; Guu et al., 2017) and reward sparsity (Agarwal et al., 2019; Muhlgay et al., 2019). In addition, semantic parsing applications only utilize the generated logical form as an intermedi- ate step in retrieving the answer. Generating logi- cal forms, however, introduces difficulties such as maintaining …

Content truncated to 500 words