An Empirical Investigation of Contextualized Number Prediction

An Empirical Investigation of Contextualized Number Prediction Daniel Spokoyny Carnegie Mellon University dspokoyn@cs.cmu.edu Taylor Berg-Kirkpatrick UC San Diego tberg@ucsd.edu Abstract We conduct a large scale empirical investiga- tion of contextualized number prediction in running text. Specifically, we consider two tasks: (1) masked number prediction – predict- ing a missing numerical value within a sen- tence, and (2) numerical anomaly detection – detecting an errorful numeric value within a sentence. We experiment with novel combi- nations of contextual encoders and output dis- tributions over the real number line. Specifi- cally, we introduce a suite of output distribu- tion parameterizations that incorporate latent variables to add expressivity and better fit the natural distribution of numeric values in run- ning text, and combine them with both recur- rent and transformer-based encoder architec- tures. We evaluate these models on two nu- meric datasets in the financial and scientific domain. Our findings show that output dis- tributions that incorporate discrete latent vari- ables and allow for multiple modes outper- form simple flow-based counterparts on all datasets, yielding more accurate numerical pre- diction and anomaly detection. We also show that our models effectively utilize textual con- text and benefit from general-purpose unsuper- vised pretraining.1 1 Introduction Pretraining large neural architectures (e.g. trans- formers (Devlin et al., 2019; Raffel et al., 2019)) on vast amounts of unlabeled data has lead to great improvements on a variety of NLP tasks. Typi- cally, such models are trained using a masked lan- guage modeling (MLM) objective and the resulting contextualized representations are finetuned for a particular downstream task like question answer- ing or sentence classification (Devlin et al., 2019; Lan et al., 2020). In this paper, we focus on a 1Code available at: https://github.com/dspoka/mnm related modeling paradigm, but a different task. Specifically, we investigate contextualized number prediction: predicting a real numeric value from its textual context using an MLM-style modeling objective. We conduct experiments on two specific variants: (1) masked number prediction (MNM), in which the goal is to predict the value of a masked number token in a sentence, and (2) numerical anomaly detection (NAD), with the goal of decid- ing whether a specific numeric value in a sentence is errorful or anomalous. In contrast with more standard MLM training setups, here we specifi- cally care about the accuracy of the trained masked conditional distributions rather than the contextual- ized representations they induce. While successful models for these tasks are themselves useful in applications like typo correction and forgery detec- tion (Chen et al., 2019), better models of numer- acy are essential for further improving downstream tasks like question answering, numerical informa- tion extraction (Mirza et al., 2017; Saha et al., 2017) or numerical fact checking (Thorne and Vlachos, 2017), as well as for processing number-heavy do- mains like financial news, technical specifications, and scientific articles. Further, systems that detect anomalous numbers in text have applications in practical domains – for example, medicine (Thim- bleby and Cairns, 2010) – where identification of numerical entry errors is critical. Our modeling …

Content truncated to 500 words