Sattiy at SemEval-2021 Task 9: An Ensemble Solution for Statement Verification and Evidence Finding with Tables

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 1255–1261 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 1255 Sattiy at SemEval-2021 Task 9: An Ensemble Solution for Statement Verification and Evidence Finding with Tables Xiaoyi Ruan, Meizhi Jin, Jian Ma, Haiqin Yang§, Lianxin Jiang, Yang Mo and Mengyuan Zhou Ping An Life Insurance Co., Ltd. Shenzhen, Guangdong province, China {RUANXIAOYI687, EX-JINMEIZHI001, MAJIAN446, JIANGLIANXIN769, MOYANG853}@pingan.com.cn § the corresponding author, email: hqyang@ieee.org Abstract Question answering from semi-structured ta- bles can be seen as a semantic parsing task and is significant and practical for pushing the boundary of natural language understand- ing. Existing research mainly focuses on un- derstanding contents from unstructured evi- dence, e.g., news, natural language sentences, and documents. The task of verification from structured evidence, such as tables, charts, and databases, is still less explored. This paper de- scribes sattiy team’s system in SemEval-2021 task 9: Statement Verification and Evidence Finding with Tables (SEM-TAB-FACT). This competition aims to verify statements and to find evidence from tables for scientific articles and to promote the proper interpretation of the surrounding article. In this paper, we exploited ensemble models of pre-trained language mod- els over tables, TaPas and TaBERT, for Task A and adjust the result based on some rules ex- tracted for Task B. Finally, in the leaderboard, we attain the F1 scores of 0.8496 and 0.7732 in Task A for the 2-way and 3-way evaluation, re- spectively, and the F1 score of 0.4856 in Task B. 1 Introduction Semantic parsing is one of the most important tasks in natural language processing. It not only needs to understand the meaning of natural language state- ments, but also needs to map them to meaningful executable queries, such as logical forms, SQL queries, and Python code (Pan et al., 2019; Lei et al., 2020; Zhu et al., 2021). Question answer- ing from semi-structured tables is usually seen as a semantic parsing task (Pasupat and Liang, 2015), where questions are translated into logical forms that can be executed against the table to retrieve the correct denotation (Zhong et al., 2017). Practically, it is significant in natural language understanding to verify whether a textual hypoth- esis is entailed or refuted by evidence (Benthem, 2008; D. et al., 1978). The verification problem has been extensively studied in different natural language tasks, such as natural language infer- ence (NLI) (Bowman et al., 2015), claim verifi- cation (Hanselowski et al., 2018), recognizing of textual entailment (RTE) (Dagan et al., 2005), and multi-model language reasoning (NLVR/NLVR2) (Suhr et al., 2018). However, existing research mainly focuses on verifying hypothesis from un- structured evidence, e.g., news, natural language sentences and documents. Research of verification under structured evidence, such as tables, charts, and databases, is still in the exploratory stage. This year, SemEval-2021 Task 9: Statement Ver- ification and Evidence Finding with Tables (SEM- TAB-FACT), aims to verify statements and find evidence from tables in scientific articles (Wang et al., 2021a). It is an important task targeting …

Content truncated to 500 words