NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3505 - 3523 May 22-27, 2022 c⃝2022 Association for Computational Linguistics NUMGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks Swaroop Mishra1 Arindam Mitra2 Neeraj Varshney1 Bhavdeep Sachdeva1 Peter Clark3 Chitta Baral1 Ashwin Kalyan3 1Arizona State University 2Microsoft Research 3Allen Institute for AI Abstract Given the ubiquitous nature of numbers in text, reasoning with numbers to perform sim- ple calculations is an important skill of AI systems. While many datasets and models have been developed to this end, state-of- the-art AI systems are brittle; failing to per- form the underlying mathematical reasoning when they appear in a slightly different sce- nario. Drawing inspiration from GLUE (Wang et al., 2018) that was proposed in the con- text of natural language understanding, we propose NUMGLUE, a multi-task benchmark that evaluates the performance of AI systems on eight different tasks, that at their core re- quire simple arithmetic understanding. We show that this benchmark is far from being solved with neural models including state-of- the-art large-scale language models perform- ing significantly worse than humans (lower by 46.4%). Further, NUMGLUE promotes sharing knowledge across tasks, especially those with limited training data as evidenced by the superior performance (average gain of 3.4% on each task) when a model is jointly trained on all the tasks as opposed to task- specific modeling. Finally, we hope that NUMGLUE will encourage systems that per- form robust and general arithmetic reasoning within language, a first step towards being able to perform more complex mathematical rea- soning1. 1 Introduction Reasoning with numbers is an important skill that occurs in various day-to-day scenarios and not surprisingly, numbers are ubiquitous in textual data. To train AI reasoning systems that can perform simple mathematical reasoning, many tasks have been proposed (Dua et al., 2019b; Ravichander et al., 2019; Koncel-Kedziorski et al., 2016). Despite these efforts, current state-of-the-art AI 1https://allenai.org/data/numglue Original Word Problem John had 5 apples. He gave 3 to Peter. How many apples does John have now? Fill In The Blanks Format John had 5 apples. He gave 3 to Peter. John has apples now. NLI Format Premise: John had 5 apples. He gave 3 apples to Peter. Hypothesis: John has 2 apples now. Does the hypothesis entail, contradict or is neutral to the premise? Comparison Format John had 5 apples. He gave 3 to Peter. Who has more apples? Figure 1: A system that can robustly perform numeric reasoning over language should be able to solve prob- lems such as the above, regardless of how the problem is posed. However, we observe existing systems are brittle; producing inconsistent solutions to such minor stylistic variations. systems are brittle and fail when problems involv- ing similar mathematical reasoning is posed in a slightly different manner. For instance, presenting a word problem in a different manner as shown in fig. 1, while hardly affecting human performance, is sufficient to confuse state-of-the-art AI systems2. This brittleness in …

Content truncated to 500 words