Continual Pre-training of Language Models for Math Problem Understanding with Syntax-Aware Memory Network

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5923 - 5933 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Continual Pre-training of Language Models for Math Problem Understanding with Syntax-Aware Memory Network Zheng Gong1,4†, Kun Zhou2,4†, Wayne Xin Zhao1,4∗, Jing Sha3,5, Shijin Wang5,6, Ji-Rong Wen1,4 1Gaoling School of Artificial Intelligence, Renmin University of China 2School of Information, Renmin University of China 3iFLYTEK Research, Hefei, Anhui, China 4Beijing Key Laboratory of Big Data Management and Analysis Methods, Beijing, China 5State Key Laboratory of Cognitive Intelligence, Hefei, Anhui, China 6AI Research(Central China), iFLYTEK, Wuhan, Hubei, China Abstract In this paper, we study how to continually pre- train language models for improving the under- standing of math problems. Specifically, we focus on solving a fundamental challenge in modeling math problems, i.e., how to fuse the semantics of textual description and formulas, which are highly different in essence. To ad- dress this issue, we propose a new approach called COMUS to continually pre-train lan- guage models for math problem understanding with syntax-aware memory network. In this approach, we first construct the math syntax graph to model the structural semantic informa- tion, by combining the parsing trees of the text and formulas, and then design the syntax-aware memory networks to deeply fuse the features from the graph and text. With the help of syntax relations, we can model the interaction between the token from the text and its semantic-related nodes within the formulas, which is helpful to capture fine-grained semantic correlations between texts and formulas. Besides, we de- vise three continual pre-training tasks to further align and fuse the representations of the text and math syntax graph. Experimental results on four tasks in the math domain demonstrate the effectiveness of our approach. Our code and data are publicly available at the link: https: //github.com/RUCAIBox/COMUS. 1 Introduction Understanding math problems via automated meth- ods is a desired machine capacity for artificial in- telligence assisted learning. Such a capacity is the key to the success of a variety of educational appli- cations, including math problem retrieval (Reusch et al., 2021), problem recommendation (Liu et al., 2018), and problem solving (Huang et al., 2020). To automatically understand math problems, it is feasible to learn computational representations † Equal contribution. This work was done when the two author were interns at iFLYTEK Research. ∗ Corresponding author, email: batmanfly@gmail.com Math Problem: Given that sin x is equal to 0.6 and x is an acute angle, find the value of sin 2𝑥 + tan ! " . Given [MATH] 0.6 of the value Root find sin sqrt divide tan Math Syntax Graph equal sin x ⋯ plus times root advcl nsubj comp obj ccomp obl nmod case det 2 2 x x Textual Description: Given that sin x is … find the value of $\sqrt { \sin{ 2x }+\tan { \frac { x }{ 2 } } }$ . Operator Tree ⋯ Figure 1: Illustration of a math problem with its textual …

Content truncated to 500 words