A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 975–984 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 975 Abstract We present ASDiv (Academia Sinica Di- verse MWP Dataset), a diverse (in terms of both language patterns and problem types) English math word problem (MWP) corpus for evaluating the capability of various MWP solvers. Existing MWP corpora for studying AI progress remain limited either in language usage patterns or in problem types. We thus present a new English MWP corpus with 2,305 MWPs that cover more text patterns and most problem types taught in elementary school. Each MWP is anno- tated with its problem type and grade level (for indicating the level of difficulty). Fur- thermore, we propose a metric to measure the lexicon usage diversity of a given MWP corpus, and demonstrate that ASDiv is more diverse than existing corpora. Experiments show that our proposed corpus reflects the true capability of MWP solvers more faith- fully. 1 Introduction Human math/science tests have been considered more suitable for evaluating AI progress than the Turing test (Clark and Etzioni, 2016). Among them, math word problems (MWPs) are frequently chosen to study natural language understanding and simulate human problem solving (Bakman, 2007; Mukherjee and Garain, 2008; Liang et al., 2016), because the answer is not a span within the given problem text that can be directly extracted. Table 1 shows a typical example of MWP, which consists of a few sentences that involve quantities. Current MWP corpora can be classified into four categories: (1) the Number Word Problem corpus (Shi et al., 2015), which contains number word problems only; (2) the Arithmetic Word Problem corpora (Hosseini et al., 2014; Roy et al., 2015), which involve the four basic arithmetic operations Math Word Problem A sandwich is priced at $0.75. A cup of pudding is priced at $0.25. Tim bought 2 sandwiches and 4 cups of pudding. How much money should Tim pay? Solution: 0.75 x 2 + 0.25 x 4 = 2.5 Table 1: A math word problem (addition, subtraction, multiplication and division) with either single-step or multi-step operations; (3) the Algebraic Word Problem corpora (Kushman et al., 2014; Koncel-Kedziorski et al., 2015; Roy and Roth, 2017; Upadhyay & Chang, 2015; Wang et al., 2017), which focus on algebraic MWPs; and (4) the Mixed-type MWP corpora (Huang et al., 2016, Ling et al., 2017, Amini et al., 2019), which are large-scale collections of either daily algebra or GRE/GMAT examination MWPs. Table 2 is a comparison of existing English MWP corpora. However, these existing corpora are either lim- ited in terms of the diversity of the associated prob- lem types (as well as lexicon usage patterns), or lacking information such as difficulty levels. For example, categories (1), (2), and (3) collect only certain types of MWPs. On the other hand, alt- hough large-scale mixed-type MWP corpora con- tain more problem types, the annotated answers or formulas are sometimes inconsistent, and the cor- responding difficulty level is usually …

Content truncated to 500 words