ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9006 - 9018 December 7-11, 2022 ©2022 Association for Computational Linguistics REASTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples Yilun Zhao1 Linyong Nan1 Zhenting Qi2 Rui Zhang3 Dragomir Radev1 1Yale University 2 Zhejiang University 3Penn State University {yilun.zhao, linyong.nan}@yale.edu Abstract Reasoning over tabular data requires both table structure understanding and a broad set of table reasoning skills. Current models with table- specific architectures and pre-training methods perform well on understanding table structures, but they still struggle with tasks that require various table reasoning skills. In this work, we develop REASTAP to show that high-level ta- ble reasoning skills can be injected into models during pre-training without a complex table- specific architecture design. We define 7 table reasoning skills, such as numerical operation, temporal comparison, and conjunction. Each reasoning skill is associated with one exam- ple generator, which synthesizes questions over semi-structured tables according to the sampled templates. We model the table pre-training task as a sequence generation task and pre- train REASTAP to generate precise answers to the synthetic examples. REASTAP is evalu- ated on four benchmarks covering three down- stream tasks including: 1) WIKISQL-WEAK and WIKITQ for Table Question Answering; 2) TABFACT for Table Fact Verification; and 3) LOGICNLG for Faithful Table-to-Text Gen- eration. Experimental results demonstrate that REASTAP achieves new state-of-the-art per- formance on all benchmarks and delivers a significant improvement on low-resource set- ting. Our code is publicly available at https: //github.com/Yale-LILY/ReasTAP. 1 Introduction Inspired by the massive success of pre-trained lan- guage models (LM) on free-form natural language (NL) tasks (Devlin et al., 2019; Dong et al., 2019; Raffel et al., 2020; Lewis et al., 2020), researchers have attempted to extend the pre-training to table data. Tables are a valuable form of data that or- ganize information in a structured way. They of- ten contain data that is organized in a more ac- cessible manner than in unstructured texts. To adapt the pre-training paradigm on structured tab- Conjunction: Q: What was the Company Name when the Industry is Oil and gas and Headquarters is China? A: National Petroleum, Sinopec Group Counting: Q: How many Headquarters have Employees was 662,575? A: 1 Numerical Comparison: (… abbreviate other reasoning skills …) REASTAP Walmart Supervise Which Company Name, with Headquarter was United States, has the 4th Profit? [HEAD] Rank | Company Name | Industry | … | Headquarters [ROW] 3 | Amazon| Retail | $ 386,064 |[ROW] 5 Sinopec Group | Oil and gas | … | 553,833 | China 3 Amazon Retail $ 386,064 $ 21,331 1,298,000 United States 7 CVS Health Healthcare $ 268,706 $ 7,179 256,500 United States 1 Walmart Retail $ 559,151 $ 13,510 2,300,000 United States 4 National Petroleum Oil and gas $ 283,958 $ 4,575 1,242,245 China 10 Volkswagen Automotive $ 253,965 $ 10,104 662,575 Germany 2 State Grid Electricity $ 386,618 $ 5,580 896,360 China 8 UnitedHealth Healthcare $ 257,141 $ 15,403 330,000 United States 6 …

Content truncated to 500 words