Annotating Derivations: A New Evaluation Strategy and Dataset for Algebra Word Problems

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 494–504, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics Annotating Derivations: A New Evaluation Strategy and Dataset for Algebra Word Problems Shyam Upadhyay1 and Ming-Wei Chang2 1University of Illinois at Urbana-Champaign, IL, USA 2Microsoft Research, Redmond, WA, USA upadhya3@illinois.edu minchang@microsoft.com Abstract We propose a new evaluation for auto- matic solvers for algebra word problems, which can identify mistakes that existing evaluations overlook. Our proposal is to evaluate such solvers using derivations, which reflect how an equation system was constructed from the word problem. To accomplish this, we develop an algorithm for checking the equivalence between two derivations, and show how derivation an- notations can be semi-automatically added to existing datasets. To make our exper- iments more comprehensive, we include the derivation annotation for DRAW-1K, a new dataset containing 1000 general alge- bra word problems. In our experiments, we found that the annotated derivations enable a more accurate evaluation of au- tomatic solvers than previously used met- rics. We release derivation annotations for over 2300 algebra word problems for fu- ture evaluations. 1 Introduction Automatically solving math reasoning problems is a long-pursued goal of AI (Newell et al., 1959; Bobrow, 1964). Recent work (Kushman et al., 2014; Shi et al., 2015; Koncel-Kedziorski et al., 2015) has focused on developing solvers for alge- bra word problems, such as the one shown in Fig- ure 1. Developing a solver for word problems can open several new avenues, especially for online education and intelligent tutoring systems (Kang et al., 2016). In addition, as solving word prob- lems requires the ability to understand and ana- lyze natural language, it serves as a good test-bed for evaluating progress towards goals of artificial intelligence (Clark and Etzioni, 2016). Am=Bn Cm + Dn = E Costs of apple and orange are in ratio 5 : 15 at the Acme Market. Mark wanted some fruits so he buys 5 apples and 5 oranges for 100 dollars. Find cost of each. 5m=15n,5m+5n=100! (m=15,n=5) ! Solution Equation System Derivation Figure 1: An algebra word problem with its solution, equa- tion system and derivation. Evaluating solvers on derivation is more reliable than evaluating on solution or equation sys- tem, as it reveals errors that other metric overlook. An automatic solver finds the solution of a given word problem by constructing a deriva- tion, consisting of an un-grounded equation system1 ({Am = Bn, Cm + Dn = E} in Figure 1) and alignments of numbers in the text to its coefficients (blue edges). The deriva- tion identifies a grounded equation system {5m = 15n, 5m + 5n = 100}, whose solution can then be generated to answer the problem. derivation precisely describes how the groundedA equation system was constructed from the word problem by the automatic solver. On the other hand, the grounded equation systems and the so- lutions are less informative, as they do not explain which span of text …

Content truncated to 500 words