Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6016 - 6031 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills Ori Yoran† Alon Talmor Jonathan Berant Tel-Aviv University {ori.yoran, alon.talmor, joberant}@cs.tau.ac.il Abstract Models pre-trained with a language model- ing objective possess ample world knowledge and language skills, but are known to struggle in tasks that require reasoning. In this work, we propose to leverage semi-structured tables, and automatically generate at scale question- paragraph pairs, where answering the question requires reasoning over multiple facts in the paragraph. We add a pre-training step over this synthetic data, which includes examples that require 16 different reasoning skills such as number comparison, conjunction, and fact com- position. To improve data efficiency, we sam- ple examples from reasoning skills where the model currently errs. We evaluate our approach on three reasoning-focused reading compre- hension datasets, and show that our model, PReasM, substantially outperforms T5, a popu- lar pre-trained encoder-decoder model. More- over, sampling examples based on model errors leads to faster training and higher performance. 1 Introduction Large pre-trained language models (LMs) (Devlin et al., 2019; Liu et al., 2019; Brown et al., 2020) have become the backbone of natural language pro- cessing in recent years. However, recent work has shown that they struggle in performing symbolic reasoning operations, such as composition or con- junction of facts (Talmor et al., 2019, 2020), numer- ical operations (Wallace et al., 2019; Hidey et al., 2020), and quantification (Warstadt et al., 2019), without substantial amounts of additional data. Past work on improving reasoning in pre-trained models has taken two flavors: (a) adding special- ized components for specific skills, like numerical and temporal reasoning (Ran et al., 2019; Gupta et al., 2020a; Khot et al., 2021; Chen et al., 2020a), or (b) generating synthetic examples at scale, for example, by using grammars or templates (Rozen †Work done while working at the Allen Institute for Arti- ficial Intelligence. Figure 1: An example table and question-context- answer triplets generated from the table as synthetic data. Each color corresponds to a different reasoning skill and colored cells are necessary to answer the ques- tion. The contexts shown are partial, such that the actual context contains the necessary information to answer the question and additional distractors. Answers are not necessarily extractive (e.g., date difference). et al., 2019; Zhao et al., 2019; Andreas, 2020; Asai and Hajishirzi, 2020; Campagna et al., 2020), and question generation models (Alberti et al., 2019; Puri et al., 2020; Bartolo et al., 2021). In this work, we take the latter approach and argue that semi-structured tables are a valuable re- source for automatic generation of training data that can endow LMs with reasoning skills. Tables can be crawled from the web at scale, and cover a wide range of domains and topics. Moreover, their structured nature makes them amenable to au- tomatic processes of data generation. Specifically, …

Content truncated to 500 words