Are NLP Models really able to Solve Simple Math Word Problems?

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094 June 6–11, 2021. ©2021 Association for Computational Linguistics 2080 Are NLP Models really able to Solve Simple Math Word Problems? Arkil Patel Satwik Bhattamishra Microsoft Research IndiaNavin Goyal arkil.patel@gmail.com, {t-satbh,navingo}@microsoft.com Abstract The problem of designing NLP solvers for math word problems (MWP) has seen sus- tained research activity and steady gains in the test accuracy. Since existing solvers achieve high performance on the benchmark datasets for elementary level MWPs contain- ing one-unknown arithmetic word problems, such problems are often considered “solved” with the bulk of research attention moving to more complex MWPs. In this paper, we re- strict our attention to English MWPs taught in grades four and lower. We provide strong ev- idence that the existing MWP solvers rely on shallow heuristics to achieve high performance on the benchmark datasets. To this end, we show that MWP solvers that do not have ac- cess to the question asked in the MWP can still solve a large fraction of MWPs. Sim- ilarly, models that treat MWPs as bag-of- words can also achieve surprisingly high accu- racy. Further, we introduce a challenge dataset, SVAMP, created by applying carefully chosen variations over examples sampled from exist- ing datasets. The best accuracy achieved by state-of-the-art models is substantially lower on SVAMP, thus showing that much remains to be done even for the simplest of the MWPs. 1 Introduction A Math Word Problem (MWP) consists of a short natural language narrative describing a state of the world and poses a question about some un- known quantities (see Table 1 for some examples). MWPs are taught in primary and higher schools. The MWP task is a type of semantic parsing task where given an MWP the goal is to generate an expression (more generally, equations), which can then be evaluated to get the answer. The task is challenging because a machine needs to extract relevant information from natural language text as well as perform mathematical reasoning to solve it. The complexity of MWPs can be measured along multiple axes, e.g., reasoning and linguistic PROBLEM: Text: Jack had 8 pens and Mary had 5 pens. Jack gave 3 pens to Mary. How many pens does Jack have now? Equation: 8 - 3 = 5 QUESTION SENSITIVITY VARIATION: Text: Jack had 8 pens and Mary had 5 pens. Jack gave 3 pens to Mary. How many pens does Mary have now? Equation: 5 + 3 = 8 REASONING ABILITY VARIATION: Text: Jack had 8 pens and Mary had 5 pens. Mary gave 3 pens to Jack. How many pens does Jack have now? Equation: 8 + 3 = 11 STRUCTURAL INVARIANCE VARIATION: Text: Jack gave 3 pens to Mary. If Jack had 8 pens and Mary had 5 pens initially, how many pens does Jack have now? Equation: 8 - 3 = 5 Table 1: Example of a Math Word Problem along with the types of …

Content truncated to 500 words