Towards Robust Numerical Question Answering: Diagnosing Numerical Capabilities of NLP Systems

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7950 - 7966 December 7-11, 2022 ©2022 Association for Computational Linguistics Towards Robust Numerical Question Answering: Diagnosing Numerical Capabilities of NLP Systems Jialiang Xu1∗, Mengyu Zhou2†, Xinyi He3*, Shi Han2, Dongmei Zhang2 1 University of Illinois at Urbana-Champaign 2 Microsoft Research 3 Xi’an Jiaotong University jx17@illinois.edu, hxyhxy@stu.xjtu.edu.cn, {mezho, shihan, dongmeiz}@microsoft.com Abstract Numerical Question Answering is the task of answering questions that require numerical ca- pabilities. Previous works introduce general adversarial attacks to Numerical Question An- swering, while not systematically exploring nu- merical capabilities specific to the topic. In this paper, we propose to conduct numerical capability diagnosis on a series of Numerical Question Answering systems and datasets. A series of numerical capabilities are highlighted, and corresponding dataset perturbations are de- signed. Empirical results indicate that exist- ing systems are severely challenged by these perturbations. E.g., Graph2Tree experienced a 53.83% absolute accuracy drop against the “Extra” perturbation on ASDiv-a, and BART experienced 13.80% accuracy drop against the “Language” perturbation on the numerical sub- set of DROP. As a counteracting approach, we also investigate the effectiveness of applying perturbations as data augmentation to relieve systems’ lack of robust numerical capabilities. With experiment analysis and empirical studies, it is demonstrated that Numerical Question An- swering with robust numerical capabilities is still to a large extent an open question. We dis- cuss future directions of Numerical Question Answering and summarize guidelines on future dataset collection and system design. 1 Introduction Numeracy is an essential part for real-world NLP applications (Sundaram et al., 2022; Thawani et al., 2021b; Sundararaman et al., 2020; Spithourakis and Riedel, 2018). Numerical QA (Question Answering) is one representative group of such number-dependent NLP tasks. E.g., Math Word Problem Solving (Zhang et al., 2020a; Miao et al., 2020; Koncel-Kedziorski et al., 2016), Discrete ∗ The contributions by Jialiang Xu and Xinyi He have been conducted and completed during their internships at Microsoft Research Asia, Beijing, China. † Corresponding author. Reasoning (Dua et al., 2019; Hu et al., 2019; Al- Negheimish et al., 2021a), Tabular Question An- swering (Zhong et al., 2017; Chen et al., 2020b; Zhu et al., 2021; Chen et al., 2021). These Numer- ical QA tasks require NLP systems to arrive at a numerical answer from the numbers in the ques- tion and context. By studying how existing NLP systems perform in these Numerical QA tasks, we could take a glimpse at what capabilities are re- quired for building NLP systems in the future. In an ad-hoc manner, a line of work revealed robustness issues of handling Numerical QA in existing NLP systems. Through adversarial at- tacks with designed dataset perturbations, number- related limitations were exposed: E.g., utilizing spurious correlation in datasets (Patel et al., 2021; Kumar et al., 2021; Al-Negheimish et al., 2021b; Pi et al., 2022b), incorrectly representing numbers (Nogueira et al., 2021; Kim et al., 2021) and fail- ing to extrapolate (Kim et al., 2021; Pal and Baral, 2021). This line of work …

Content truncated to 500 words