Question Directed Graph Attention Network for Numerical Reasoning over Text

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6759–6768, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6759 Question Directed Graph Attention Network for Numerical Reasoning over Text Kunlong Chen† Weidi Xu† Xingyi Cheng∗† Zou Xiaochuan† Yuyu Zhang§ Le Song†§ Taifeng Wang† Yuan Qi† Wei Chu† † Ant Group {kunlong.ckl,weidi.xwd,fanyin.cxy,xiaochuan.zxc, le.song,taifeng.wang,weichu.cw,yuan.qi}@antgroup.com § College of Computing Georgia Institute of Technology {yuyu}@gatech.edu Abstract Numerical reasoning over texts, such as ad- dition, subtraction, sorting and counting, is a challenging machine reading comprehension task, since it requires both natural language un- derstanding and arithmetic computation. To address this challenge, we propose a heteroge- neous graph representation for the context of the passage and question needed for such rea- soning, and design a question directed graph attention network to drive multi-step numerical reasoning over this context graph. Our model, which combines deep learning and graph rea- soning, achieves remarkable results in bench- mark datasets such as DROP 1. 1 Introduction Machine reading comprehension (MRC) aims to de- velop AI models that can answer questions for text doc- uments. Recently, the performance of MRC in public datasets has been improved dramatically due to the ad- vanced pre-trained models, such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019) and ALBERT (Lan et al., 2019). However, pre-trained models are not explicitly aware of the concepts of numerical reasoning since numer- acy supervision signals are rarely available during pre- training. The representations from these pre-trained models fall short in their ability to support downstream numerical reasoning. Yet such ability is critical for the comprehension of financial news and scientific articles, since basic numerical operations, such as addition, sub- traction, sorting and counting, need to be conducted to extract the essential information (Dua et al., 2019). Recently, Dua et al. (2019) proposed a numerically- aware QANet (NAQANet), which treats the span ex- tractions, counting, and numerical addition/subtraction separately. However, this work is preliminary in the sense that the model neglects the relative magnitude between numbers. To improve this method, Ran et al. ∗Corresponding author 1https://leaderboard.allenai.org/drop/submissions/public. As of September 08, 2020, our models are ranked first in the case of fair comparison using the identical pre-training model. (2019) proposed NumNet, which constructs a number comparison graph that encodes the relative magnitude information between numbers on directed edges. Al- though NumNet achieves superior performance than other numerically-aware models (Hu et al., 2019a; An- dor et al., 2019; Geva et al., 2020; Chen et al., 2020), we argue that NumNet is insufficient for sophisticated nu- merical reasoning, since it lacks two critical ingredients for numerical reasoning: 1. Number Type and Entity Mention. The number comparison graph in NumNet is not able to identify different number types, and lacks the information of entities mentioned in the document that connect the number nodes. 2. Direct Interaction with Question. The graph rea- soning module in NumNet leaves out the direct ques- tion representation, which may encounter difficulties in locating important numbers directed by the ques- tion as the …

Content truncated to 500 words