LILA: A Unified Benchmark for Mathematical Reasoning

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5807 - 5832 December 7-11, 2022 ©2022 Association for Computational Linguistics L¯ILA: A Unified Benchmark for Mathematical Reasoning Swaroop Mishra∗ † Arizona State Univeristy Matthew Finlayson∗ ‡ The Allen Institute for AI Pan Lu† UCLA Leonard Tang Harvard University Sean Welleck The Allen Institute for AI Chitta Baral Arizona State Univeristy Tanmay Rajpurohit Georgia Institute of Technology Oyvind Tafjord The Allen Institute for AI Ashish Sabharwal The Allen Institute for AI Peter Clark The Allen Institute for AI Ashwin Kalyan‡ The Allen Institute for AI Abstract Mathematical reasoning skills are essential for general-purpose intelligent systems to perform tasks from grocery shopping to climate mod- eling. Towards evaluating and improving AI systems in this domain, we propose L¯ILA, a unified mathematical reasoning benchmark consisting of 23 diverse tasks along four dimen- sions: (i) mathematical abilities e.g., arithmetic, calculus (ii) language format e.g., question- answering, fill-in-the-blanks (iii) language di- versity e.g., no language, simple language (iv) external knowledge e.g., commonsense, physics. We construct our benchmark by ex- tending 20 datasets benchmark by collecting task instructions and solutions in the form of Python programs, thereby obtaining explain- able solutions in addition to the correct an- swer. We additionally introduce two evalu- ation datasets to measure out-of-distribution performance and robustness to language pertur- bation. Finally, we introduce BH ¯ASKARA, a general-purpose mathematical reasoning model trained on L¯ILA. Importantly, we find that multi-tasking leads to significant improvements (average relative improvement of 21.83% F1 score vs. single-task models), while the best performing model only obtains 60.40%, indi- cating the room for improvement in general mathematical reasoning and understanding.1 1 Introduction Mathematical reasoning is required in all aspects of life, from buying ingredients for a recipe to con- ∗Equal first authors. †Work done while at the Allen Institute for AI. ‡Corresponding authors: matthewf@allenai.org, ash- winkv@allenai.org. 1Our dataset: https://github.com/allenai/Lila. Our model: https://huggingface.co/allenai/bhaskara. ��������������������������������������������������������� ��������������������������������������������������������� ����������� ������������������� ������������������� ����������������� ���������������������������������������������������� ������������������������� ����������� ������ ������ ������������������������������������������������������ �������������� ������������� ���������� ������������������������ ������������������������������������ ������������������������������������� �������������������������������� �������������������������������������������������������� ������������������������������������������������������ ����������������������������������������������������������� ���������������������������������� Figure 1: A data example with two Python programs in L¯ILA. One program annotation uses function construct whereas the other one is a plain script without function. The instruction for each task and categories across four dimensions are annotated for developing L¯ILA. trolling the world economy. Given the fundamen- tal nature of mathematical reasoning, a number of works propose datasets to evaluate specific mathe- matical reasoning abilities of AI agents, e.g., Kush- man et al. (2014) (algebra word problems), Mishra et al. (2022c) (arithmetic reasoning), Saxton et al. (2019) (templated math reasoning spanning alge- bra, calculus, probability, etc.) Since evaluating high-capacity models on narrowly scoped mathe- matical reasoning datasets risks overestimating the reasoning abilities of these AI systems, creating the need for a unified benchmark for systematic evaluation over diverse topics and problem styles. 5807 To this end, we introduce L¯ILA2, a unified math- ematical reasoning benchmark that consists of 23 mathematical reasoning tasks. L¯ILA is constructed by extending 20 …

Content truncated to 500 words