How well do Computers Solve Math Word Problems? Large-Scale Dataset Construction and Evaluation

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 887–896, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics How Well Do Computers Solve Math Word Problems? Large-Scale Dataset Construction and Evaluation Danqing Huang1∗, Shuming Shi2, Chin-Yew Lin2, Jian Yin1 and Wei-Ying Ma2 1 Sun Yat-sen University {huangdq2@mail2,issjyin@mail}.sysu.edu.cn 2 Microsoft Research {shumings,cyl,wyma}@microsoft.com Abstract Recently a few systems for automatically solving math word problems have reported promising results. However, the datasets used for evaluation have limitations in both scale and diversity. In this paper, we build a large-scale dataset which is more than 9 times the size of previous ones, and contains many more problem types. Problems in the dataset are semi- automatically obtained from community question-answering (CQA) web pages. A ranking SVM model is trained to auto- matically extract problem answers from the answer text provided by CQA users, which significantly reduces human anno- tation cost. Experiments conducted on the new dataset lead to interesting and surpris- ing results. 1 Introduction Designing computer systems for automatically solving math word problems is a challenging re- search topic that dates back to the 1960s (Bobrow, 1964a; Briars and Larkin, 1984; Fletcher, 1985). As early proposals seldom report empirical evalu- ation results, it is unclear how well they perform. Recently, promising results have been reported on both statistical learning approaches (Kushman et al., 2014; Hosseini et al., 2014; Koncel-Kedziorski et al., 2015; Zhou et al., 2015; Roy and Roth, 2015) and semantic parsing methods (Shi et al., 2015). However, we observe two limitations on the datasets used by these previous works. First, the datasets are small. The most frequently used dataset (referred to as Alg514 hereafter) only con- tains 514 algebra problems. The Dolphin1878 ∗Work done while this author was an intern at Microsoft Research. dataset (Shi et al., 2015), the largest collection among them, contains 1878 problems. Second, the diversity of problems in the datasets is low. The Alg514 collection contains linear algebra prob- lems of 28 types (determined by 28 unique equa- tion systems), with each problem type correspond- ing to at least 6 problems. Although the Dol- phin1878 collection has over 1,000 problem types, only number word problems (i.e., math word prob- lems about the operations and relationship of num- bers) are contained in the collection. Due to the above two limitations, observations and conclusions based on existing datasets may not be representative. Therefore it is hard to give a convincing answer to the following question: How well do state-of-the-art computer algorithms per- form in solving math word problems? To answer this question, we need to re-evaluate state-of-the-art approaches on a larger and more diverse data set. It is not hard to collect a large set of problems from the web. The real challenge comes from attaching annotations to the problems. Important annotation types include equation sys- tems (required by most statistical learning meth- ods for model training) and gold answers (for testing algorithm performance). Manually adding equation systems and gold answers is extremely time-consuming1. …

Content truncated to 500 words