TABBIE: Pretrained Representations of Tabular Data

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3446–3456 June 6–11, 2021. ©2021 Association for Computational Linguistics 3446 TABBIE: Pretrained Representations of Tabular Data Hiroshi Iida† Dung Thai‡ Varun Manjunatha§ Mohit Iyyer‡ †Sony Corporation ‡UMass Amherst hiroshi.iida@sony.com§Adobe Research {dthai,miyyer}@cs.umass.edu vmanjuna@adobe.com Abstract Existing work on tabular representation- learning jointly models tables and associated text using self-supervised objective functions derived from pretrained language models such as BERT. While this joint pretraining improves tasks involving paired tables and text (e.g., an- swering questions about tables), we show that it underperforms on tasks that operate over tables without any associated text (e.g., pop- ulating missing cells). We devise a simple pretraining objective (corrupt cell detection) that learns exclusively from tabular data and reaches the state-of-the-art on a suite of table- based prediction tasks. Unlike competing ap- proaches, our model (TABBIE) provides em- beddings of all table substructures (cells, rows, and columns), and it also requires far less com- pute to train. A qualitative analysis of our model’s learned cell, column, and row repre- sentations shows that it understands complex table semantics and numerical trends. 1 Introduction Large-scale self-supervised pretraining has sub- stantially advanced the state-of-the-art in natural language processing (Peters et al., 2018; Devlin et al., 2018; Liu et al., 2019). More recently, these pretraining methods have been extended to jointly learn representations of tables as well as text (Herzig et al., 2020; Yin et al., 2020), which enables improved modeling of tasks such as ques- tion answering over tables. However, many prac- tical problems involve semantic understanding of tabular data without additional text-based input, such as extracting tables from documents, retriev- ing similar columns or cells, and filling in miss- ing information (Zhang and Balog, 2020). In this work, we design a pretraining methodology specifi- cally for tables (Tabular Information Embedding or TABBIE) that resembles several core tasks in table extraction and decomposition pipelines and 5 Spain 3.6 Medals Size France Italy 4 step 1: corrupt 15% of cells TABBIE real real corrupt! real real corrupt! real real step 2: embed the table with TABBIE step 3: train TABBIE to identify the corrupted cells Figure 1: TABBIE is a table embedding model trained to detect corrupted cells, inspired by the ELEC- TRA (Clark et al., 2020) objective function. This sim- ple pretraining objective results in powerful embed- dings of cells, columns, and rows, and it yields state- of-the-art results on downstream table-based tasks. allows easy access to representations for different tabular substructures (cells, rows, and columns). Existing table representation models such as TaBERT (Yin et al., 2020) and TaPas (Herzig et al., 2020) concatenate tabular data with an associated piece of text and then use BERT’s masked lan- guage modeling objective for pretraining. These approaches are computationally expensive due to the long sequences that arise from concatenating text with linearized tables, which necessitates trun- cating the input sequences1 to make training fea- sible. We show that TaBERT underperforms on downstream table-based …

Content truncated to 500 words