Volta at SemEval-2021 Task 9: Statement Verification and Evidence Finding with Tables using TAPAS and Transfer Learning

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 1262–1270 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 1262 Volta at SemEval-2021 Task 9: Statement Verification and Evidence Finding with Tables using TAPAS and Transfer Learning Devansh Gautam∗ Kshitij Gupta∗ Manish Shrivastava International Institute of Information Technology Hyderabad {devansh.gautam,kshitij.gupta}@research.iiit.ac.in, m.shrivastava@iiit.ac.in Abstract Tables are widely used in various kinds of doc- uments to present information concisely. Un- derstanding tables is a challenging problem that requires an understanding of language and table structure, along with numerical and logi- cal reasoning. In this paper, we present our sys- tems to solve Task 9 of SemEval-2021: State- ment Verification and Evidence Finding with Tables (SEM-TAB-FACTS). The task con- sists of two subtasks: (A) Given a table and a statement, predicting whether the table sup- ports the statement and (B) Predicting which cells in the table provide evidence for/against the statement. We fine-tune TAPAS (a model which extends BERT’s architecture to capture tabular structure) for both the subtasks as it has shown state-of-the-art performance in various table understanding tasks. In subtask A, we evaluate how transfer learning and standard- izing tables to have a single header row im- proves TAPAS’ performance. In subtask B, we evaluate how different fine-tuning strate- gies can improve TAPAS’ performance. Our systems achieve an F1 score of 67.34 in sub- task A three-way classification, 72.89 in sub- task A two-way classification, and 62.95 in subtask B. 1 Introduction There has been extensive work on verifying if a given textual context supports a given statement. Even though tables are also widely used to con- vey information, especially in scientific texts, there has been comparatively less work on verifying if a given table supports a statement. To this end, Se- mEval 2021 Task 9 (Wang et al., 2021) focuses on statement verification and evidence finding for ta- bles from scientific articles in the English language. The task is divided into two subtasks - A and B. The aim of subtask A is to classify whether a given ∗The authors have contributed equally. statement is entailed or refuted according to the given table and associated table metadata (such as captions and legends) or whether the statement’s truth is unknown as it cannot be determined from the table. The aim of subtask B is to classify each cell in the table as relevant or irrelevant in deter- mining whether the statement is entailed or refuted from the tabular evidence (the truth value of the statement is also provided). Our systems use TAPAS (Herzig et al., 2020) trained with intermediate pre-training (Eisenschlos et al., 2020) for both the subtasks. For subtask A, we fine-tune TAPAS after adding a three-way clas- sification head on top for classifying the statement as entailed/refuted/unknown. We also evaluate how transfer learning and standardizing tables to have a single header row can improve TAPAS’ perfor- mance. Due to the similarity between subtask B and table question-answering (which involves cell selection or cell selection followed by aggregation), …

Content truncated to 500 words