Representing Numbers in NLP: a Survey and a Vision

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–656 June 6–11, 2021. ©2021 Association for Computational Linguistics 644 Representing Numbers in NLP: a Survey and a Vision Avijit Thawani and Jay Pujara and Filip Ilievski and Pedro Szekely University of Southern California Information Sciences Institute {thawani,jpujara,ilievski,pszekely}@isi.edu Abstract NLP systems rarely give special consideration to numbers found in text. This starkly con- trasts with the consensus in neuroscience that, in the brain, numbers are represented differ- ently from words. We arrange recent NLP work on numeracy into a comprehensive tax- onomy of tasks and methods. We break down the subjective notion of numeracy into 7 sub- tasks, arranged along two dimensions: gran- ularity (exact vs approximate) and units (ab- stract vs grounded). We analyze the myriad representational choices made by over a dozen previously published number encoders and de- coders. We synthesize best practices for repre- senting numbers in text and articulate a vision for holistic numeracy in NLP, comprised of de- sign trade-offs and a unified evaluation. 1 Introduction Numbers are an integral part of text. To under- stand a simple sentence like I woke up at 11, we need not just literacy but also numeracy. We must decode the string 11 to the quantity 11 and infer 11 to denote a time of the day, probably 11 a.m. We need commonsense to reason that 11 a.m. is quite late in the morning. This interpretation of 11 is strongly contextual, as I earn $11 per month evokes different units and value expectations. Note how the semantics remains the same for both sen- tences if 11 was replaced by 10, i.e., the context is tolerant to some variability. Numbers are everywhere. Reasoning with quantities and counts is crucial to understanding the world. Evolutionary learning has given numer- ical cognition skills to several animals, including human beings (Dehaene, 2011). Our ancient an- cestors furthered numeracy by developing multiple number systems, similar to but independent from the evolution of languages. Numeracy is an essen- tial skill for language understanding, since numbers are often interspersed in text: the 6 million pages in English Wikipedia have over 150 million numbers. Numbers are neglected. In NLP, however, numbers are either filtered out explicitly during preprocessing (Graff et al., 2003), or treated the same as words, often collapsing them into an UNK token. Subword tokenization approaches like BPE (Sennrich et al., 2016) and WordPiece (Wu et al., 2016) instead retain numbers, but split them into arbitrary tokens, for example 1234 might be split into two tokens as 12-34 or 123-4 or 1-234. Recent work has shown that these are subopti- mal number representations (Wallace et al., 2019; Zhang et al., 2020). On the DROP Question An- swering benchmark, BERT performs five times worse when the answer is a number instead of a span of text (Dua et al., 2019). Relatively simple strategies like switching from subword to char-level tokenization (Geva et al., 2020), or from …

Content truncated to 500 words