Multitasking Framework for Unsupervised Simple Definition Generation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 528 - 537 May 22-27, 2022 c⃝2022 Association for Computational Linguistics TABLEFORMER: Robust Transformer Modeling for Table-Text Encoding Jingfeng Yang ∗ Aditya Gupta† Shyam Upadhyay† Luheng He† Rahul Goel† Shachi Paul † ⋆Georgia Institute of Technology †Google Assistant jingfengyangpku@gmail.com tableformer@google.com Abstract Understanding tables is an important aspect of natural language understanding. Existing mod- els for table understanding require lineariza- tion of the table structure, where row or col- umn order is encoded as an unwanted bias. Such spurious biases make the model vulner- able to row and column order perturbations. Additionally, prior work has not thoroughly modeled the table structures or table-text align- ments, hindering the table-text understanding ability. In this work, we propose a robust and structurally aware table-text encoding architec- ture TABLEFORMER, where tabular structural biases are incorporated completely through learnable attention biases. TABLEFORMER is (1) strictly invariant to row and column or- ders, and, (2) could understand tables better due to its tabular inductive biases. Our eval- uations showed that TABLEFORMER outper- forms strong baselines in all settings on SQA, WTQ and TABFACT table reasoning datasets, and achieves state-of-the-art performance on SQA, especially when facing answer-invariant row and column order perturbations (6% im- provement over the best baseline), because pre- vious SOTA models’ performance drops by 4% - 6% when facing such perturbations while TABLEFORMER is not affected.1 1 Introduction Recently, semi-structured data (e.g. variable length tables without a fixed data schema) has attracted more attention because of its ubiquitous presence on the web. On a wide range of various table rea- soning tasks, Transformer based architecture along with pretraining has shown to perform well (Eisen- schlos et al., 2021; Liu et al., 2021). In a nutshell, prior work used the Transformer architecture in a BERT like fashion by serializing tables or rows into word sequences (Yu et al., 2020; ∗Work done during an internship at Google. 1Code to be released at https://github.com/ google-research/tapas Title Length Screwed Up 5:02 Ghetto Queen 5:00 Question: Of all song lengths, which one is the longest? Gold Answer: 5:02 TAPAS: 5:00 TAPAS after row order perturbation: 5:02 TABLEFORMER: 5:02 (a) TAPAS predicts incorrect answer based on the original table, while it gives the correct answer if the first row is moved to the end of the table. Nation Gold Silver Bronze Great Britain 2 1 2 Spain 1 2 0 Ukraine 0 2 0 Question: Which nation received 2 silver medals? Gold Answer: Spain, Ukraine TAPAS: Spain TABLEFORMER: Spain, Ukraine TABLEFORMER w/o a proposed structural bias: Spain (b) TAPAS gives incomplete answer due to its limited cell grounding ability. Figure 1: Examples showing the limitations of exist- ing models (a) vulnerable to perturbations, and (b) lack- ing structural biases. In contrast, our proposed TABLE- FORMER predicts correct answers for both questions. Liu et al., 2021), where original position ids are used as positional information. Due to the usage of row/column ids and global position …

Content truncated to 500 words