FORTAP: Using Formulas for Numerical-Reasoning-Aware Table Pretraining

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1150 - 1166 May 22-27, 2022 c⃝2022 Association for Computational Linguistics FORTAP: Using Formulas for Numerical-Reasoning-Aware Table Pretraining Zhoujun Cheng1∗, Haoyu Dong2∗ † , Ran Jia2, Pengfei Wu3, Shi Han2, Fan Cheng1† , Dongmei Zhang2 1MoE Key Laboratory of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai 200240, China 2Microsoft Research Asia, China 3Fudan University, China {blankcheng,chengfan}@sjtu.edu.cn, 17307130207@fudan.edu.cn {hadong,raji,shihan,dongmeiz}@microsoft.com Abstract Tables store rich numerical data, but numerical reasoning over tables is still a challenge. In this paper, we find that the spreadsheet formula, a commonly used language to perform computa- tions on numerical values in spreadsheets, is valuable supervision for numerical reasoning in tables. Considering large amounts of spreadsheets available on the web, we propose FORTAP , the first exploration to leverage spreadsheet formulas for table pretraining. Two novel self-supervised pretraining objec- tives are derived from formulas, numerical reference prediction (NRP) and numerical calculation prediction (NCP). While our proposed objectives are generic for encoders, to better capture spreadsheet table layouts and structures, we build FORTAP upon TUTA, the first transformer-based method for spread- sheet&web table pretraining with tree attention. FORTAP outperforms state-of-the-art methods by large margins on three representative datasets of formula prediction, question an- swering, and cell type classification, showing the great potential of leveraging formulas for table pretraining. The code will be released at https://github.com/microsoft/TUTA_ table_understanding. 1 Introduction Tables store rich numerical data, so a wide range of tasks require numerical reasoning over (semi- )structured tabular context, such as question an- swering over tables (Chen et al., 2021b; Zhu et al., 2021; Cheng et al., 2021), table-to-text (Suadaa et al., 2021; Moosavi et al., 2021; Cheng et al., 2021), spreadsheet formula prediction (Chen et al., 2021a), and table structure understanding (Koci et al., 2019). Take Table#2 in Figure 1 as an example, both suggesting the formula (C4-B4)/B4 for cell D4 and answering “0.61%” to the question require ∗The first two authors contribute equally. †Corresponding authors. =(D3 – C3) / C3 % Increase references corresponding numerical values in 2016 and 2021. % Increase involves compositional calculations of subtraction and division. Large scale pretraining FORTAP: FORmula-driven TAble Pretraining •Formula suggestion: D4=(C4-B4)/B4 Table#1 with formulae for self-supervised pretraining Table#2 with/without formula •Table structure understanding: Matrix table with a derived %Change column and a derived Country row. •Table-to-text: Belgium's population increased by 0.61% in 2020 compared to 2019. Downstream task finetuning •Question answering: What percentage of Belgium's population has increased in 2020 compared to 2019? -- 0.61% Figure 1: It’s desirable to learn numerical reasoning via formula pretraining and generalize it to various tasks. numerical reasoning capabilities of (1) understand- ing the contextual meaning of individual numerical cells, e.g., “11.49” at B4 and “11.56” at C4 are “population”s of “Belgium” in “2019” and “2020”; (2) inferring calculational relationships of numeri- cal cells, e.g., percentage change from “11.49” to “11.56”. As Figure 1 shows, same capabilities also benefit table structure recognition and table-to-text. So it’s …

Content truncated to 500 words