MATE: Multi-view Attention for Table Transformer Efficiency

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7606–7619 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7606 MATE: Multi-view Attention for Table Transformer Efficiency Julian Martin Eisenschlos1, Maharshi Gor2∗, Thomas Müller3∗, William W. Cohen1 Google Research1 {eisenjulian,wcohen}@google.com Dept. of Computer Science, University of Maryland2 mgor@cs.umd.edu Symanto Research, Valencia, Spain3 thomas.mueller@symanto.com Abstract This work presents a sparse-attention Trans- former architecture for modeling documents that contain large tables. Tables are ubiquitous on the web, and are rich in information. How- ever, more than 20% of relational tables on the web have 20 or more rows (Cafarella et al., 2008), and these large tables present a chal- lenge for current Transformer models, which are typically limited to 512 tokens. Here we propose MATE, a novel Transformer architec- ture designed to model the structure of web ta- bles. MATE uses sparse attention in a way that allows heads to efficiently attend to either rows or columns in a table. This architecture scales linearly with respect to speed and memory, and can handle documents containing more than 8000 tokens with current accelerators. MATE also has a more appropriate inductive bias for tabular data, and sets a new state-of-the-art for three table reasoning datasets. For HY- BRIDQA (Chen et al., 2020b), a dataset that in- volves large documents containing tables, we improve the best prior result by 19 points. 1 Introduction The Transformer architecture (Vaswani et al., 2017) is expensive to train and run at scale, especially for long sequences, due to the quadratic asymptotic complexity of self-attention. Although some work addresses this limitation (Ainslie et al., 2020; Ki- taev et al., 2020; Zaheer et al., 2020), there has been little prior work on scalable Transformer architec- tures for semi-structured text.1 However, although some of the more widely used benchmark tasks in- volving semi-structured data have been restricted to moderate size tables, many semi-structured docu- ments are large: more than 20% of relational tables on the web have 20 or more rows (Cafarella et al., ∗Work done at Google Research. 1The term "semi-structured text" refers to text that has structure that does not reflect a known data schema. Typically semi-structured text is organized as an HTML tree or variable length lists and tables. Pick # Player College 27 Tom Wilfrid Laurier 28 Mark York 29 Pedro California 30 Rob York ... ... ... Query: Which college got the most players? Column Head Row Head Figure 1: Sparse self-attention heads on tables in MATE are of two classes: Row heads attend to tokens inside cells in the same row, as well as the query. Column heads attend to tokens in the same column and in the query. Query tokens attend to all other tokens. 2008), and would pose a problem for typical Trans- former models. Here we study how efficient implementations for transformers can be tailored to semi-structured data. Figure 1 highlights our main motivation through an example: to obtain a contextual representation of a cell in a table, it is …

Content truncated to 500 words