Overlooked Data in Typological Databases: What Grambank Teaches Us About Gaps in Grammars

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 291–297 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 291 Evaluating Transformer Language Models on Arithmetic Operations Using Number Decomposition Matteo Muffo, Aldo Cocco, Enrico Bertino Indigo.ai Via Torino 61, Milan, Italy {matteo, aldo, e}@indigo.ai Abstract In recent years, Large Language Models such as GPT-3 showed remarkable capabilities in performing NLP tasks in the zero and few shot settings. On the other hand, the experiments highlighted the difficulty of GPT-3 in carrying out tasks that require a certain degree of reasoning, such as arithmetic operations. In this paper we evaluate the ability of Transformer Language Models to perform arithmetic operations following a pipeline that, before performing computations, decomposes numbers in units, tens, and so on. We denote the models fine-tuned with this pipeline with the name Calculon and we test them in the task of performing additions, subtractions and multiplications on the same test sets of GPT-3. Results show an increase of accuracy of 63% in the five-digit addition task. Moreover, we demonstrate the importance of the decomposition pipeline introduced, since fine-tuning the same Language Model without decomposing numbers results in 0% accuracy in the five-digit addition task. Keywords: Transformer Language Models, arithmetic operations, number decomposition 1. Introduction The publication of GPT-3 (Brown et al., 2020) had a relevant impact on Natural Language Processing, showing that it is possible to leverage a Large Lan- guage Model (LLM) to perform downstream tasks in the zero and few shot setting. However, although GPT- 3 showed on-the-fly reasoning capabilities in tasks such as two or three-digit operations, it struggles with five- digit operations. This suggests that a LLM such as GPT-3 did not effectively learn to perform arithmetic operations and is not able to generalize its ability to perform sums or subtractions to any number of digits. With this work we want to assess if Transformer Lan- guage Models have enough reasoning capabilities to learn to perform arithmetic operations of unseen num- bers. To do so, we introduce Calculon, a GPT-2 (Rad- ford et al., 2019) model fine-tuned to perform arith- metic operations between decomposed numbers. In particular, Calculon is trained to perform arithmetic op- erations following a pipeline that decomposes the num- bers in digit form (e.g. 18954 = 4 units, 5 tens, 9 hun- dreds, 8 thousands, 1 tens of thousands). The under- lying idea is to teach LMs to do computations as chil- dren learn at school, processing units with units, tens with tens, and so on. Following this pipeline, Calculon reaches remarkable levels of accuracy, even in the four and five-digit tasks where GPT-3 obtains poor results. To validate the importance of the pipeline here pro- posed, we fine-tune a GPT-2 model on the same datasets of Calculon without decomposing numbers. In this setting, the fine-tuned GPT-2 network reaches very poor performances on the four and five digits tasks, demonstrating that the decomposition pipeline is a valid approach to make LMs …

Content truncated to 500 words