Table Fact Verification with Structure-Aware Transformer

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1624–1629, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1624 Table Fact Verification with Structure-Aware Transformer∗ Hongzhi Zhang†, Yingyao Wang⋄, Sirui Wang†, Xuezhi Cao†, Fuzheng Zhang†, Zhongyuan Wang† † Meituan Dianping Group, Beijing, China ⋄Harbin Institute of Technology, China {zhanghongzhi03,wangsirui,caoxuezhi,zhangfuzheng}@meituan.com yywang@hit-mtlab.net, wzhy@outlook.com Abstract Verifying fact on semi-structured evidence like tables requires the ability to encode structural information and perform symbolic reasoning. Pre-trained language models trained on natu- ral language could not be directly applied to encode tables, because simply linearizing ta- bles into sequences will lose the cell alignment information. To better utilize pre-trained trans- formers for table representation, we propose a Structure-Aware Transformer (SAT), which injects the table structural information into the mask of the self-attention layer. A method to combine symbolic and linguistic reasoning is also explored for this task. Our method outper- forms baseline with 4.93% on TabFact, a large scale table verification dataset. 1 Introduction Table fact verification aims at classifying whether a textual hypothesis is entailed or refuted by the given table. It could benefit downstream tasks such as fake news detection, misinformation de- tection, etc. Compared to fact verification over tex- tual evidence (Dagan et al., 2006; Bowman et al., 2015), verification on semi-structured data further requires 1) the ability to encode and understand structural information of tables, and 2) the abil- ity to perform symbolic reasoning over structured data, such as counting, comparing, and numeri- cal calculation. Although large-scale pre-trained language models (Devlin et al., 2019; Yang et al., 2019) achieved dominant results on textual entail- ment datasets (Wang et al., 2019), they could not be directly used to encode semi-structured data as they are pre-trained on unstructured natural language. Wenhu et al. (2020) eliminate the discrepancy by serializing tables into word sequences, and then table fact verification could be processed as a natu- ral language inference task. The most straightfor- ward method for table serialization is linearizing ∗The first two authors contribute equally to this work. the table contents via horizontal scan. However, this would destroy structural information within tables, i.e. the alignments between table cells. In Figure 1, the value “533” and “733” is meaning- less digits without the column name “core clock”, and it is hard for the model to recover the align- ments from the flattened word sequence. Therefore, Table-BERT (Wenhu et al., 2020) includes the col- umn name into cell representation using natural language templates during the linearization. How- ever, comparing or counting column contents of different rows over the flattened word sequence remains a hard task, and simply duplicating the column name multiple times does not achieve sat- isfying results. To better utilize the transformer architecture for table representation, we propose to inject the ta- ble’s structural information into the mask of the self-attention layer. Figure 2 illustrates the pattern commonly adopted when human read or write a table. Usually, each table row describes a record, and cell c1,2 describes a record property with …

Content truncated to 500 words