Analyzing ChatGPT’s Mathematical Deficiencies: Insights and Contributions

Analyzing ChatGPT’s Mathematical Deficiencies: Insights and Contributions Vincent Cheng Morrison Academy vincentcheng236@gmail.com Zhang Yu National Central University phoenix000.taipei@gmail.com Abstract In this study, we assess ChatGPT, OpenAI’s latest conversational chatbot and large lan- guage model (LLM), on its performance in elementary-grade arithmetic and logic prob- lems. Despite its impressive coherence in nat- ural language processing and ability to follow instructions, our findings indicate that Chat- GPT still has room for improvement in math- ematical tasks. To evaluate its performance, we used six math and logic datasets, including SingleEq, AddSub, SVAMP, MultiArith, Sim- ple Arithmetic and counting, and Arithmetic (word variation), and found that ChatGPT per- formed better than previous models such as InstructGPT and Minerva. However, our arith- metic dataset, which includes two- to seven- digit equations, revealed that ChatGPT’s accu- racy in solving addition problems decreased from 100% to 64%, with simple arithmetic er- rors such as not carrying over in addition being a common issue. Additionally, the model strug- gled with basic multi-step word problems. To address this, we propose a novel benchmark for evaluating LLMs’ mathematical abilities. Fur- ther research is needed for LLMs to reach the level of mathematical reasoning comparable to their natural language processing abilities. Overall, our study highlights the need for con- tinued improvement in LLMs’ mathematical abilities to make them more effective in real- world applications. Keywords: Large language models, reasoning capabilities 1 Introduction Pretrained language models (PLMs) have revolu- tionized natural language processing, achieving im- pressive performance on various tasks, from senti- ment analysis to question answering and text gener- ation. With the development of large language mod- els (LLMs), the capabilities of PLMs have grown even further, with models such as GPT-3 boasting over 100 billion parameters [Brown et al., 2020]. ChatGPT, a conversational chatbot and LLM de- veloped by OpenAI, has become one of the most popular language models, with over 100 million users in under three months. However, while these models excel in language processing, they may lack the ability to reason mathematically and logically, as observed in previous models such as BART [Pa- tel et al., 2021, Wang et al., 2021, Roy and Roth, 2016]. In this paper, we present a study of the mathe- matical and logical capabilities of ChatGPT, focus- ing on simple arithmetic, elementary-grade level math word problems, and logic problems. While previous research has analyzed ChatGPT’s perfor- mance on advanced math problems with proofs from college-level pure math courses [Frieder et al., 2023], our research is unique as it presents a de- tailed analysis of ChatGPT’s performance on sim- ple mathematical and logical reasoning tasks. Our study evaluates ChatGPT’s mathematical reasoning abilities, which have not been analyzed in previous research. Moreover, while Borji [2023] briefly touches on various topics such as mathematical reasoning, hallucination, and bias, our analysis focuses solely on the model’s ability to reason mathematically and logically. Our research aims to specifically contribute to the evaluation of LLMs’ mathematical and logical capabilities. Our study makes several contributions to the evaluation of ChatGPT’s …

Content truncated to 500 words