Numeracy enhances the Literacy of Language Models

Numeracy enhances the Literacy of Language Models Avijit Thawani Jay Pujara University of Southern California / Los Angeles Information Sciences Institute / Marina del Rey {thawani,jpujara,ilievski}@isi.edu Filip Ilievski Abstract Specialized number representations in NLP have shown improvements on numerical rea- soning tasks like arithmetic word problems and masked number prediction. But humans also use numeracy to make better sense of world concepts, e.g., you can seat 5 people in your room but not 500. Does a better grasp of numbers improve a model’s understanding of other concepts and words? This paper stud- ies the effect of using six different number en- coders on the task of masked word prediction (MWP), as a proxy for evaluating literacy. To support this investigation, we develop Wiki- Convert, a 900,000 sentence dataset annotated with numbers and units, to avoid conflating nominal and ordinal number occurrences. We find a significant improvement in MWP for sentences containing numbers, that exponent embeddings are the best number encoders, yielding over 2 points jump in prediction ac- curacy over a BERT baseline, and that these enhanced literacy skills also generalize to con- texts without annotated numbers. We release all code at https://git.io/JuZXn. 1 Introduction Numbers account for 6.15% of all unique tokens in English Wikipedia (Jiang et al., 2020), yet NLP systems have traditionally either removed numbers during preprocessing or replaced them with a sin- gle uninformative UNK token. Recent models such as BERT retain them but learn individual token embeddings for hundreds of numbers. Moreover, subword tokenization approaches end up segment- ing numbers into possibly suboptimal splits, e.g., 4500 is seen as (4, 500) or (45, 00) depending on the specific tokenizer used. The human brain, in contrast, automatically maps numbers to their approximate magnitude on the number line (Dehaene, 2011). NLP sys- tems that fail to account for the scalar values that numbers denote may correspondingly lack in com- prehension. Recent work has empirically demon- strated the inefficacy of existing NLP methods in numeric reasoning tasks (Wallace et al., 2019). Al- ternative number representations have been pro- posed, such as projecting the number’s magnitude into a vector space (Sundararaman et al., 2020) or switching to a scientific notation (Zhang et al., 2020; Berg-Kirkpatrick and Spokoyny, 2020). BERT Exp The [mask] weighs 100 lbs. statue bomb The [mask] weighs 10000 lbs. statue car Table 1: Numerate language models perform better at masked word prediction. BERT: Default BERT base- line. Exp: BERT with exponent embeddings (§2). We observe that this line of work goes from literacy to numeracy, i.e., helping language mod- els gain numerate skills such as simple arith- metic (Geva et al., 2020), measurement estimation (Zhang et al., 2020), and masked number predic- tion (Berg-Kirkpatrick and Spokoyny, 2020). Our work addresses the converse question: Do alterna- tive number representations enhance the ability of language models to understand/predict words? We investigate this question through experiments with several representative number encoders, pro- posed in prior work. We develop and release Wiki- Convert, a large, novel dataset of number-annotated sentences, …

Content truncated to 500 words