Have You Seen That Number? Investigating Extrapolation in Question Answering Models

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7031–7037 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7031 Have You Seen That Number? Investigating Extrapolation in Question Answering Models Jeonghwan Kim Giwon Hong∗ Kyung-min Kim∗ Junmo Kang∗ Sung-Hyon Myaeng School of Computing, KAIST Daejeon, Republic of Korea {jeonghwankim123, giwon.hong, kimdarwin, junmo.kang, myaeng}@kaist.ac.kr Abstract Numerical reasoning in machine reading com- prehension (MRC) has shown drastic improve- ments over the past few years. While the pre- vious models for numerical MRC are able to interpolate the learned numerical reasoning ca- pabilities, it is not clear whether they can per- form just as well on numbers unseen in the training dataset. Our work rigorously tests state-of-the-art models on DROP, a numerical MRC dataset, to see if they can handle pas- sages that contain out-of-range numbers. One of the key findings is that the models fail to ex- trapolate to unseen numbers. Presenting num- bers as digit-by-digit input to the model, we also propose the E-digit number form that al- leviates the lack of extrapolation in models and reveals the need to treat numbers differ- ently from regular words in the text. Our work provides a valuable insight into the numerical MRC models and the way to represent number forms in MRC. 1 Introduction The research in question-answering (QA) models that are able to perform reading comprehension and discrete reasoning over numbers in the passage has seen significant progress, like the models in DROP (Ran et al., 2019; Hu et al., 2019; Chen et al., 2020; Geva et al., 2020). Despite their abil- ity to understand the complex context and num- bers within, none of them deal with the notion of whether these models can robustly handle "unseen" numbers during testing. The ability to extend dis- crete, symbolic rules such as addition and subtrac- tion on numbers outside what we already know is called extrapolation, and this is an essential part of human intelligence. For example, if one can reason on numbers over text that range between 0 and 100, it is logically reasonable to infer that it should be able to handle numbers larger than ∗Equal contribution. As of the census of 2000, there were 49,927 people, 18,009 households, and 12,192 families residing in the county. The population density was 48 people per square mile (19/km²). As of the census of 2000, there were 4,992,700 people, 1,800,900 households, and 1,219,200 families residing in the county. The population density was 4,800 people per square mile (19/km²). “How many people, households, and families reside in the county according to the 2000 census?” Answer: 80,128 Answer: 8,012,800 Figure 1: An example of DROP data instance perturba- tion to test the numerical QA models for extrapolation capabilities. This figure shows CARDINAL-type num- bers factored by 100. 100. The lack of extrapolation capability in models, however, is a significant obstacle in the way toward a truly generalizable, number-understanding QA. Although the problem of numerical extrapola- tion has been recently addressed by previous works …

Content truncated to 500 words