SemEval-2021 Task 9: Fact Verification and Evidence Finding for Tabular Data in Scientific Documents (SEM-TAB-FACTS)

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 317–326 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 317 SemEval-2021 Task 9: Fact Verification and Evidence Finding for Tabular Data in Scientific Documents (SEM-TAB-FACTS) Nancy X. R. Wang∗ Diwakar Mahajan∗ Marina Danilevsky Sara Rosenthal§ IBM Research nancywang1991@gmail.com, {dmahaja, mdanile, sjrosenthal}@us.ibm.com Abstract Understanding tables is an important and rel- evant task that involves understanding table structure as well as being able to compare and contrast information within cells. In this pa- per, we address this challenge by presenting a new dataset and tasks that addresses this goal in a shared task in SemEval 2020 Task 9: Fact Verification and Evidence Finding for Tabu- lar Data in Scientific Documents (SEM-TAB- FACTS). Our dataset contains 981 manually- generated tables and an auto-generated dataset of 1980 tables providing over 180K statement and over 16M evidence annotations. SEM- TAB-FACTS featured two sub-tasks. In sub- task A, the goal was to determine if a statement is supported, refuted or unknown in relation to a table. In sub-task B, the focus was on iden- tifying the specific cells of a table that provide evidence for the statement. 69 teams signed up to participate in the task with 19 success- ful submissions to subtask A and 12 successful submissions to subtask B. We present our re- sults and main findings from the competition. 1 Introduction Tables are ubiquitous in documents and presenta- tions for conveying important information in a con- cise manner. This is true in many domains, stretch- ing from scientific to government documents. In fact, surrounding text in these articles are often statements summarizing or highlighting some in- formation derived from the primary source of data in tables. A relevant example is shown in Figure 1 from a Business Insider article analyzing the im- pact of Covid-19 (Aylin Woodward and Gal, 2020). Describing all the information provided in this ta- ble in a readable manner would be lengthy and considerably more difficult to understand. Despite their importance, popular question answering (e.g. SQuAD and Natural Question (Rajpurkar et al., ∗ Equal Contribution § Corresponding Author Figure 1: Surrounding text often highlights some infor- mation from the table but does not capture all data. Al- ternately, the linked text may be subjective or even mis- leading without the original table to check the claims. 2016; Kwiatkowski et al., 2019)) and truth verifi- cation tasks (e.g. SemEval-2019 Fact Checking Task (Mihaylova et al., 2019)) have not focused on tables, being composed solely of written text. This is likely due to their complexity to parse and under- stand, despite their rich amount of information. Further, the structure of tables allows much more information to be presented in an efficient manner as humans can interpret meaning in the spatial re- lationship between cells. However, due to their challenging nature, recent algorithms have been less successful at extracting (Hoffswell and Liu, 2019) and understanding header and data structure in tables (Cafarella et al., 2018). In addition, any …

Content truncated to 500 words