A Survey of Deep Learning for Mathematical Reasoning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14605–14631 July 9-14, 2023 ©2023 Association for Computational Linguistics A Survey of Deep Learning for Mathematical Reasoning Pan Lu1, Liang Qiu1, Wenhao Yu2, Sean Welleck3∗, Kai-Wei Chang1∗ 1UCLA, 2University of Notre Dame, 3University of Washington https://github.com/lupantech/dl4math Abstract Mathematical reasoning is a fundamental as- pect of human intelligence and is applicable in various fields, including science, engineering, finance, and everyday life. The development of artificial intelligence (AI) systems capable of solving math problems and proving theorems in language has garnered significant interest in the fields of machine learning and natural language processing. For example, mathemat- ics serves as a testbed for aspects of reasoning that are challenging for powerful deep learning models, driving new algorithmic and model- ing advances. On the other hand, recent ad- vances in large-scale neural language models have opened up new benchmarks and oppor- tunities to use deep learning for mathematical reasoning. In this survey paper, we review the key tasks, datasets, and methods at the inter- section of mathematical reasoning and deep learning over the past decade. We also evaluate existing benchmarks and methods, and discuss future research directions in this domain. 1 Introduction “The study of mathematics, like the Nile, begins in minuteness but ends in magnificence.” — Charles Caleb Colton, English writer Mathematical reasoning is a key aspect of hu- man intelligence that enables us to comprehend and make decisions based on numerical data and lan- guage. It is applicable in various fields, including science, engineering, finance, and everyday life, and encompasses a range of abilities, from basic skills such as pattern recognition and numerical operations to more advanced skills like problem- solving, logical reasoning, and abstract thinking. The development of artificial intelligence (AI) sys- tems capable of solving math problems and proving theorems in language has been a long-standing fo- cus of research in the fields of machine learning and ∗denotes co-senior authors. natural language processing (NLP), dating back to the 1960s (Feigenbaum et al., 1963; Bobrow, 1964). In recent years, there has been a surge of interest in this area: for instance, the number of papers has grown from approximately 10 in 2018 to 66 in 2022 (see Figure 3 in the Appendix). As deep learning continues to revolutionize NLP tasks such as question answering and machine translation (Sutskever et al., 2014; Devlin et al., 2019), it has also made significant strides in the field of mathematical reasoning (Wang et al., 2017; Yang and Deng, 2019; Geva et al., 2020; Wei et al., 2022). However, despite the impressive capabilities of these models, there is still a lack of a clear tax- onomy of the different types of mathematical rea- soning tasks and the specific capabilities required of deep learning models to solve them. Previous literature has been limited to the dis- cussion of specific aspects, such as solving math word problems (Bhattacharya, 2017; Zhang et al., 2019; Ughade and Kumbhar, 2019), representing numbers representation (Thawani …

Content truncated to 500 words