FERMAT: An Alternative to Accuracy for Numerical Reasoning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15026–15043 July 9-14, 2023 ©2023 Association for Computational Linguistics FERMAT: An Alternative to Accuracy for Numerical Reasoning Jasivan Alex Sivakumar and Nafise Sadat Moosavi Department of Computer Science University of Sheffield United Kingdom {jasivakumar1|n.s.moosavi}@sheffield.ac.uk Abstract While pre-trained language models achieve im- pressive performance on various NLP bench- marks, they still struggle with tasks that re- quire numerical reasoning. Recent advances in improving numerical reasoning are mostly achieved using very large language models that contain billions of parameters and are not ac- cessible to everyone. In addition, numerical reasoning is measured using a single score on existing datasets. As a result, we do not have a clear understanding of the strengths and short- comings of existing models on different numer- ical reasoning aspects and therefore, potential ways to improve them apart from scaling them up. Inspired by CheckList (Ribeiro et al., 2020), we introduce a multi-view evaluation set for nu- merical reasoning in English, called FERMAT. Instead of reporting a single score on a whole dataset, FERMAT evaluates models on various key numerical reasoning aspects such as num- ber understanding, mathematical operations, and training dependency. Apart from providing a comprehensive evaluation of models on dif- ferent numerical reasoning aspects, FERMAT enables a systematic and automated generation of an arbitrarily large training or evaluation set for each aspect.The datasets and codes are pub- licly available to generate further multi-view data for ulterior tasks and languages.1 1 Introduction Numerical reasoning is an aspect that is often for- gotten despite being an integral part of natural language. It is the ability to interact with num- bers using the fundamental mathematical prop- erties and thus model an area of human cogni- tive thinking (Saxton et al., 2019). Better under- standing of numbers in language models would benefit various tasks like fact-checking (Vlachos and Riedel, 2015), text generation (Moosavi et al., 2021; Suadaa et al., 2021), and educational tools 1https://github.com/jasivan/FERMAT (Mandal et al., 2022). Current models’ perfor- mance are still too weak with respect to numer- ical accuracy to then be used in downstream tasks like Infotabs (Gupta et al., 2020) which requires identifying numbers in tables and then perform- ing operations to correctly label statements causing factuality errors in such tasks. Recently, we have observed improved perfor- mances on relevant datasets about numerical rea- soning using very large language models (Wei et al., 2022b; Lewkowycz et al., 2022; Kojima et al., 2022). However, there are two main limitations to this recent trend. First, as models become larger their access becomes restricted to fewer users, i.e., users with the computational resources of large companies. For example, using one of the best mathematical models, the 540B parameter model Minerva (Lewkowycz et al., 2022), would require over 2212G of memory for inference only. Sec- ond, the numerical reasoning capabilities of ex- isting models are measured using a single score, i.e., mostly accuracy on common benchmarks like GSM8K (Cobbe et al., 2021). …

Content truncated to 500 words