NumNet: Machine Reading Comprehension with Numerical Reasoning

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2474–2484, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2474 NumNet: Machine Reading Comprehension with Numerical Reasoning Qiu Ran1∗, Yankai Lin1∗, Peng Li1, Jie Zhou1, Zhiyuan Liu2 1Pattern Recognition Center, WeChat AI, Tencent Inc, China 2Department of Computer Science and Technology, Tsinghua University, Beijing, China Institute for Artificial Intelligence, Tsinghua University, Beijing, China State Key Lab on Intelligent Technology and Systems, Tsinghua University, Beijing, China {soulcaptran,yankailin,patrickpli,withtomzhou}@tencent.com liuzy@tsinghua.edu.cn Abstract Numerical reasoning, such as addition, sub- traction, sorting and counting is a critical skill in human’s reading comprehension, which has not been well considered in existing machine reading comprehension (MRC) systems. To address this issue, we propose a numerical MRC model named as NumNet, which uti- lizes a numerically-aware graph neural net- work to consider the comparing information and performs numerical reasoning over num- bers in the question and passage. Our sys- tem achieves an EM-score of 64.56% on the DROP dataset, outperforming all existing ma- chine reading comprehension models by con- sidering the numerical relations among num- bers. 1 Introduction Machine reading comprehension (MRC) aims to infer the answer to a question given the docu- ment. In recent years, researchers have proposed lots of MRC models (Chen et al., 2016; Dhingra et al., 2017; Cui et al., 2017; Seo et al., 2017) and these models have achieved remarkable results in various public benchmarks such as SQuAD (Ra- jpurkar et al., 2016) and RACE (Lai et al., 2017). The success of these models is due to two rea- sons: (1) Multi-layer architectures which allow these models to read the document and the ques- tion iteratively for reasoning; (2) Attention mech- anisms which would enable these models to focus on the part related to the question in the document. However, most of existing MRC models are still weak in numerical reasoning such as addi- tion, subtraction, sorting and counting (Dua et al., 2019), which are naturally required when read- ing financial news, scientific articles, etc. Dua et al. (2019) proposed a numerically-aware QANet ∗indicates equal contribution (NAQANet) model, which divides the answer gen- eration for numerical MRC into three types: (1) extracting spans; (2) counting; (3) addition or sub- traction over numbers. NAQANet makes a pio- neering attempt to answer numerical questions but still does not explicitly consider numerical reason- ing. To tackle this problem, we introduce a novel model NumNet that integrates numerical reason- ing into existing MRC models. A key problem to answer questions requiring numerical reasoning is how to perform numerical comparison in MRC systems, which is crucial for two common types of questions: (1) Numerical Comparison: The answers of the questions can be directly obtained via perform- ing numerical comparison, such as sorting and comparison, in the documents. For example, in Table 1, for the first question, if the MRC system knows the fact that “49 > 47 > 36 > 31 > …

Content truncated to 500 words