Numerical reasoning in machine reading comprehension tasks: are we there yet?

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9643–9649 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9643 Numerical reasoning in machine reading comprehension tasks: are we there yet? Hadeel Al-Negheimish1, Pranava Madhyastha2,1 and Alessandra Russo1 1 Imperial College London 2 City, University of London {halnegheimish,a.russo}@imperial.ac.uk, pranava.madhyastha@city.ac.uk Abstract Numerical reasoning based machine reading comprehension is a task that involves reading comprehension along with using arithmetic op- erations such as addition, subtraction, sorting, and counting. The DROP benchmark (Dua et al., 2019) is a recent dataset that has inspired the design of NLP models aimed at solving this task. The current standings of these mod- els in the DROP leaderboard, over standard metrics, suggest that the models have achieved near-human performance. However, does this mean that these models have learned to rea- son? In this paper, we present a controlled study on some of the top-performing model architectures for the task of numerical reason- ing. Our observations suggest that the standard metrics are incapable of measuring progress to- wards such tasks. 1 Introduction Machine reading comprehension (MRC) primarily involves building automated models that are capa- ble of answering arbitrary natural language ques- tions on a given textual context such as a paragraph. Solving this problem should in principle require a fine-grained understanding of the question and comprehension of the textual context to arrive at the correct answer. Designing an MRC benchmark is challenging, as it is easy to inadvertently craft questions that allow models to exploit cues that al- low them to bypass the intended reasoning required (Gardner et al., 2019). Recent advances in NLP, currently dominated by transformer-based pre-trained models, have re- sulted in models that indicate, when measured with standard metrics, human-like performance on a va- riety of benchmarks over leaderboards for MRC1. In this paper, we focus on numerical reasoning based MRC and investigate DROP (Dua et al., 2019), a recent benchmark designed to measure 1for eg., https://leaderboard.allenai.org/ complex multi-hop and discrete reasoning, includ- ing numerical reasoning2. In contrast to single- span extraction tasks, DROP allows sets of spans, numbers, and dates as possible answers. We are particularly interested in numerical questions, for eg., ‘How many years did it take for the population to decrease to about 1100 from 10000?’ which requires extracting the corresponding years for the associated populations from the given passage, fol- lowed by computing the time difference in years. The benchmark has inspired the design of spe- cialized BERT and embedding-based NLP mod- els aimed at solving this task, seemingly achiev- ing near-human performance (evaluated using F1 scores) as reported in the DROP leaderboard. In this work, we investigate some of DROP’s top-performing models on the leaderboard in or- der to understand the extent to which these models are capable of performing numerical reasoning, in contrast to relying on spurious cues. We probe the models with a variety of perturbation techniques to assess how well models understand the ques- tion, and to what extent such models are basing the …

Content truncated to 500 words