Arithmetic-Based Pretraining Improving Numeracy of Pretrained Language Models

Proceedings of the The 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 477–493 July 13-14, 2023 ©2023 Association for Computational Linguistics Arithmetic-Based Pretraining – Improving Numeracy of Pretrained Language Models Dominic Petrak† , Nafise Sadat Moosavi‡, Iryna Gurevych† †Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science and Hessian Center for AI (hessian.AI), Technical University of Darmstadt, Germany https://www.ukp.tu-darmstadt.de ‡Department of Computer Science, The University of Sheffield, UK Abstract State-of-the-art pretrained language models tend to perform below their capabilities when applied out-of-the-box on tasks that require un- derstanding and working with numbers. Recent work suggests two main reasons for this: (1) popular tokenisation algorithms have limited expressiveness for numbers, and (2) common pretraining objectives do not target numeracy. Approaches that address these shortcomings usually require architectural changes or pre- training from scratch. In this paper, we propose a new extended pretraining approach called Arithmetic-Based Pretraining that jointly ad- dresses both in one extended pretraining step without requiring architectural changes or pre- training from scratch. Arithmetic-Based Pre- training combines contrastive learning to im- prove the number representation, and a novel extended pretraining objective called Inferable Number Prediction Task to improve numer- acy. Our experiments show the effectiveness of Arithmetic-Based Pretraining in three differ- ent tasks that require improved numeracy, i.e., reading comprehension in the DROP dataset, inference-on-tables in the InfoTabs dataset, and table-to-text generation in the WikiBio and Sci- Gen datasets1. 1 Introduction Numbers are ubiquitous in natural language. There- fore, understanding and working with numbers (usually referred to as numeracy) is a critical ca- pability for pretrained language models such as BART (Lewis et al., 2020) or T5 (Raffel et al., 2019), cornerstones of modern NLP, in order to utilize quantitative information for various NLP tasks. Recent works question whether these mod- els meet this requirement out-of-the-box (Wallace et al., 2019; Zhang et al., 2020): Common pretrain- ing objectives such as the denoising autoencoder of 1Code, data, and models trained using Arithmetic- Based Pretraining are available here: https://github.com/ UKPLab/starsem2023-arithmetic-based-pretraining. BART (Lewis et al., 2020), the masked language modeling objective of BERT (Devlin et al., 2019), or the span-corruption objective of T5 (Raffel et al., 2019), are designed for understanding structure and semantic meaning of language and not to learn working with numbers. Furthermore, commonly used subword-based tokenisation algorithms such as Byte Pair Encoding (Sennrich et al., 2016) or WordPiece (Wu et al., 2016) are designed to handle patterns that are frequently observed during train- ing, which is disadvantageous for numbers. For instance, 0.72 and 0.73 are two similar numbers. They should be processed similarly, but according to their frequency in the pretraining data they might be tokenised very differently, e.g., [0, ., 72] and [0, ., 7, 3], which will have an impact on their repre- sentation in embedding space. To address these shortcomings, various approaches have been pro- posed recently. However, most of them introduce additional components or rely on predefined fea- tures that limit their application, e.g., they are only applicable in a specific …

Content truncated to 500 words