Structural Encoding and Pre-training Matter: Adapting BERT for Table-Based Fact Verification

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2366–2375 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2366 Structural Encoding and Pre-training Matter: Adapting BERT for Table-Based Fact Verification Rui Dong David A. Smith Khoury College of Computer Sciences Northeastern University {dongrui, dasmith}@ccs.neu.edu Abstract Growing concern with online misinformation has encouraged NLP research on fact veri- fication. Since writers often base their as- sertions on structured data, we focus here on verifying textual statements given evi- dence in tables. Starting from the Table Parsing (TAPAS) model developed for ques- tion answering (Herzig et al., 2020), we find that modeling table structure improves a language model pre-trained on unstructured text. Pre-training language models on En- glish Wikipedia table data further improves performance. Pre-training on a question an- swering task with column-level cell rank infor- mation achieves the best performance. With improved pre-training and cell embeddings, this approach outperforms the state-of-the-art Numerically-aware Graph Neural Network ta- ble fact verification model (GNN-TabFact), increasing statement classification accuracy from 72.2% to 73.9% even without model- ing numerical information. Incorporating nu- merical information with cell rankings and pre-training on a question-answering task in- creases accuracy to 76%. We further ana- lyze accuracy on statements implicating sin- gle rows or multiple rows and columns of tables, on different numerical reasoning sub- tasks, and on generalizing to detecting errors in statements derived from the ToTTo table-to- text generation dataset. 1 Introduction The rapid growth in the amount and sources of on- line textual content has raised concerns about mis- information and its potential harmful impacts on society when quickly spread to a massive audience. For example, a study on enhancing medical edu- cation with Wikipedia in 2015 found that 97% of medical students completing the survey disclosed that they found mistakes in Wikipedia medical en- tries (Herbert et al., 2015). Concerns about mis- information have stimulated extensive research on automatic fact verification, i.e., verifying whether a given textual statement is entailed or refuted by the given evidence. Figure 1: Example from the TabFact dataset: the top table contains the structured evidence; the bottom two boxes contain the statements entailed and refuted by the evidence. Errors are highlighted in red. While most existing fact verification work fo- cuses on unstructured textual evidence, fact ver- ification with structured evidence is still under- explored. Recently, Chen et al. (2019) introduced a new large-scale dataset, TabFact, for verifying statements based on structured evidence in tables. Figure 1 presents an example table and the cor- responding entailed and refuted statements from TabFact. The task of fact verification based on structured evidence is challenging in two aspects. First, traditional language models trained on un- structured text are not directly applicable to learn representations for structured text. It is difficult for such language models to understand a sentence like “Round Pick Player College 1 1 Ralph Samp- son Virginia ...” by directly concatenating the ta- ble cells in Figure 1. Second, detecting misinfor- mation with …

Content truncated to 500 words