TAPAS at SemEval-2021 Task 9: Reasoning over tables with intermediate pre-training

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 423–430 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 423 TAPAS at SemEval-2021 Task 9: Reasoning over tables with intermediate pre-training Thomas Müller, Julian Martin Eisenschlos, Syrine Krichene Google Research, Zürich {thomasmueller,eisenjulian,syrinekrichene}@google.com Abstract We present the TAPAS contribution to the Shared Task on Statement Verification and Ev- idence Finding with Tables (SemEval 2021 Task 9, Wang et al. (2021)). SEMTABFACT Task A is a classification task of recognising if a statement is entailed, neutral or refuted by the content of a given table. We adopt the bi- nary TAPAS model of Eisenschlos et al. (2020) to this task. We learn two binary classification models: A first model to predict if a statement is neutral or non-neutral and a second one to predict if it is entailed or refuted. As the shared task training set contains only entailed or re- futed examples, we generate artificial neutral examples to train the first model. Both models are pre-trained using a MASKLM objective, in- termediate counter-factual and synthetic data (Eisenschlos et al., 2020) and TABFACT (Chen et al., 2020), a large table entailment dataset. We find that the artificial neutral examples are somewhat effective at training the first model, achieving 68.03 test F1 versus the 60.47 of a majority baseline. For the second stage, we find that the pre-training on the intermediate data and TABFACT improves the results over MASKLM pre-training (68.03 vs 57.01). 1 Introduction Recently, the task of Textual Entailment (TE) (Da- gan et al., 2005) or Natural Language Inference (NLI) (Bowman et al., 2015) has been adapted to a setup where the premise is a table (Chen et al., 2020; Gupta et al., 2020). The Shared Task on Statement Verification and Evidence Finding with Tables (SemEval 2021 Task 9, Wang et al. (2021)) follows this line of work and provides a new dataset consisting of tables extracted from scientific arti- cles and natural language statements written by crowd workers. In this paper, we discuss a system for tackling task A, which is a multi-class classi- fication task that requires finding if a statement is Step 1: Pretrain with MLM [CLS] Statement [SEP] Header 1 [MASK] [MASK] ... Step 2: Intermediate Pretrain with Counterfactual+Synthetic data Sum of wins when Country is U.S is 7 → Entailed Greg Steve has the highest earnings → Refuted Step 4a: Fine-tune neutral detector Real SemTabFact Instance → Relevant Corrupted SemTabFact Instance → Neutral Step 4b: Fine-tune binary entailment Real SemTabFact Instance → Entailed / Refuted Step 3: Intermediate Pretrain on TabFact Real TabFact Instance → Entailed / Refuted Figure 1: Overview of the training pipeline use in our system. We use intermediate pre-training on Counter- factual+Synthetic data (Eisenschlos et al., 2020) and then fine-tune on TABFACT (Chen et al., 2020). entailed, neutral or refuted by the contents of a table. The training set contains only entailed and refuted examples and requires data augmentation to learn the neutral class. Additionally, this data set …

Content truncated to 500 words