Joint Word and Morpheme Segmentation with Bayesian Non-Parametric Models

Findings of the Association for Computational Linguistics: EACL 2023, pages 69–77 May 2-6, 2023 ©2023 Association for Computational Linguistics Improving Numeracy by Input Reframing and Quantitative Pre-Finetuning Task Chung-Chi Chen,1 Hiroya Takamura,2 Ichiro Kobayashi,3 Yusuke Miyao2 1 Artificial Intelligence Research Center, AIST, Japan 2 Ochanomizu University, Japan 3 University of Tokyo, Japan c.c.chen@acm.com, takamura.hiroya@aist.go.jp, koba@is.ocha.ac.jp, yusuke@is.s.u-tokyo.ac.jp Abstract Numbers have unique characteristics to words. Teaching models to understand numbers in text is an open-ended research question. Instead of discussing the required calculation skills, this paper focuses on a more fundamental topic: un- derstanding numerals. We point out that innu- meracy—the inability to handle basic numeral concepts—exists in most pretrained language models (LMs), and we propose a method to solve this issue by exploring the notation of numbers. Further, we discuss whether chang- ing notation and pre-finetuning along with the comparing-number task can improve perfor- mance in three benchmark datasets contain- ing quantitative-related tasks. The results of this study indicate that input reframing and the proposed pre-finetuning task is useful for RoBERTa. 1 Introduction Numerals are an indispensable part of narratives and provide much fine-grained information.1 How models learn the number system has intrigued many researchers (Spithourakis and Riedel, 2018; Naik et al., 2019; Chen et al., 2019; Wallace et al., 2019; Zhang et al., 2020). Researchers have long dis- cussed some numeracy-related properties of pre- trained language models (LMs). In this study, we propose a new concept — innumeracy. The prob- lem of innumeracy becomes most evident when models are faced with numerals that do not appear in training data, e.g., when the range of numerals in training data is different from that in the test data. Moreover, LMs often face difficulties un- derstanding numbers even though the numbers are present in the training data. One possible cause of this problem is that numerals can have various notations, some of which are difficult to understand from their subwords. Another possible cause is 1In this paper, we focus on the numerals represented by digits (0 to 9 and decimal point) and do not discuss those written in words such as “one” and “two”. Model Notation Tokenized Example BERT Org. "147", "##70", "##2" Digit "1", "4", "7", "7", "0", "2" SN "1", ".", "47", "##70", "##200", "##00", "##0", "##e", "+", "05" RoBERTa Org. "147", "702" Digit "1", "4", "7", "7", "0", "2" SN "1", ".", "47", "70", "200000", "E", "+", "05" Table 1: Tokenized example. Org. and SN denote original and scientific notation, respectively. that LMs are not pretrained to deal with numbers. Therefore, in this study, we address the problem of innumeracy via input reframing and quantitative pre-finetuning tasks. Input reframing refers to changing the notations of numbers, which can be one of the crucial clues for understanding numerals (Zhang et al., 2020; Chen et al., 2021). In addition to the original notation, we consider the digit-based and scien- tific notations. Table 1 lists examples of using different representations for numerals. Our exper- iments indicate that RoBERTa (Liu et al., 2019) performs …

Content truncated to 500 words