THiFly_Queens at SemEval-2021 Task 9: Two-stage Statement Verification with Adaptive Ensembling and Slot-based Operation

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 416–422 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 416 THiFly Queens at SemEval-2021 Task 9: Two-stage Statement Verification with Adaptive Ensembling and Slot-based Operation Yuxuan Zhou1, Kaiyin Zhou2,3, Xien Liu1, Ji Wu1, Xiaodan Zhu4 1Department of Electronic Engineering, Tsinghua University, Beijing 100084, China 2THiFLY Research, Tsinghua University, Beijing 100084, China 3State Key Laboratory of Cognitive Intelligence, Hefei, Anhui 230088, China 4ECE & Ingenuity Labs Research Institute, Queen’s University zhouyx17@mails.tsinghua.edu.cn kyzhou3@iflytek.com {xeliu, wuji ee}@mail.tsinghua.edu.cn xiaodan.zhu@queensu.ca Abstract This paper describes our system for verifying 9.statements with tables at SemEval-2021 Task We developed a two-stage verifying sys- tem based on the latest table-based pre-trained model GraPPa. Multiple networks are de- vised to verify different types of statements in the competition dataset and an adaptive model ensembling technique is applied to ensemble models in both stages. A statement-slot-based symbolic operation module is also used in our system to further improve the performance and stability of the system. Our model achieves second place in the 3-way classification and fourth place in the 2-way classification evalu- ation. Several ablation experiments show the effectiveness of different modules proposed in this paper. 1 Introduction Verifying whether a statement is entailed, refuted, or unknown with the given table is a challenging task, which requires the system to understand the statement and table jointly and reasons from the two information resources. The studies on this task could benefit several downstream applications, e.g. fake news detection. Recently, with the release of a large-scale dataset for table-based fact verification named TABFACT (Chen et al., 2019), this task re- ceived several studies. Verifying statements based on tables is a challenging task since the researches on understanding tables are not enough compared with works on free-texts, and the methods to train models understanding free-text and table jointly need further studies as well. While TABFACT contains a huge number of state- ments and tables, the tables in the TABFACT dataset are relatively simple since they do not have hierarchical column heads as tables in scientific papers do and the contents in the tables are easier to understand compared to the tables in scientific papers as well. In the SemEval task 9 (Wang et al., 2021), the goal is to develop a system that can verify statements (subtask A) and find evidence (subtask B) based on the tables extracted from sci- entific tables. Our team is more interested in the verifying task and only participated in subtask A. Different from data in TABFACT, in subtask A, we are also required to classify a new type of state- ment that cannot be entailed or refuted based on the given table, named “unknown” type. Since no statements of this type are given in the training data, the classification of this type of statement becomes a core difficulty of the subtask. This paper describes our two-stage table-based ver- ifying system based on the latest table-based pre- trained language model GraPPa (Yu et al., 2020). …

Content truncated to 500 words