Improving Math Word Problems with Pre-trained Knowledge and Hierarchical Reasoning

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3384–3394 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3384 Improving Math Word Problems with Pre-trained Knowledge and Hierarchical Reasoning Weijiang Yu, Yingpeng Wen, Fudan Zheng, Nong Xiao∗ School of Computer Science and Engineering, Sun Yat-sen University weijiangyu8@gmail.com, {wenyp6,zhengfd3}@mail2.sysu.edu.cn, xiaon6@mail.sysu.edu.cn Abstract The recent algorithms for math word prob- lems (MWP) neglect to use outside knowl- edge not present in the problems. Most of them only capture the word-level relation- ship and ignore to build hierarchical reason- ing like the human being for mining the contextual structure between words and sen- tences. In this paper, we propose a Reasoning with Pre-trained Knowledge and Hierarchical Structure (RPKHS) network, which contains a pre-trained knowledge encoder and a hier- archical reasoning encoder. Firstly, our pre- trained knowledge encoder aims at reasoning the MWP by using outside knowledge from the pre-trained transformer-based models. Sec- ondly, the hierarchical reasoning encoder is presented for seamlessly integrating the word- level and sentence-level reasoning to bridge the entity and context domain on MWP. Exten- sive experiments show that our RPKHS signifi- cantly outperforms state-of-the-art approaches on two large-scale commonly-used datasets, and boosts performance from 77.4% to 83.9% on Math23K, from 75.5 to 82.2% on Math23K with 5-fold cross-validation and from 83.7% to 89.8% on MAWPS. More extensive ablations are shown to demonstrate the effectiveness and interpretability of our proposed method. 1 Introduction Math Word Problem (MWP) is a reasoning task for answering a mathematical query based on the problem description, which is an interdisciplinary research topic to bridge the mathematics and nat- ural language processing. As shown in Table 1, a short narrative is presented to describe a problem and poses a question about the unknown quantity. In recent years, research on MWP by using deep learning methods has been gaining increasing at- tention. Early research mainly focuses on Seq2Seq- based models (Sutskever et al., 2014; Ling et al., ∗Corresponding Author Problem: Conner has 25000 dollars in his bank account. Every month he spends 1500 dollars. He does not add money to the account. How much money will Conner have in his account after 8 months? Equation: x = 25000.0 − (1500.0 ∗ 8.0); Solution: 13000.0 Table 1: The example of the math word problem task. Given a natural language description for a mathemat- ical problem, it requires the model to infer a formal math equation and final quantity solution. 2017b; Wang et al., 2017; Huang et al., 2018; Wang et al., 2017). These Seq2Seq-based methods aim to train an end-to-end model from scratch by using the training dataset. Some research focuses on de- veloping structure-based approaches (Xie and Sun, 2019a; Wang et al., 2018a, 2019b; Liu et al., 2019a; Zhang et al., 2020b; Li et al., 2020b; Hong et al., 2021; Li et al., 2020a) by incorporating parsing tree into the neural models to produce promising results in generating solution expression for the MWP. To answer this question, human beings not only need to parse …

Content truncated to 500 words