Measuring Forecasting Skill from Text

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5317–5331 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5317 Measuring Forecasting Skill from Text Shi Zong1 Alan Ritter1 Eduard Hovy2 1Department of Computer Science and Engineering, The Ohio State University 2Language Technologies Institute, Carnegie Mellon University {zong.56, ritter.1492}@osu.edu, hovy@cs.cmu.edu Abstract People vary in their ability to make accurate predictions about the future. Prior studies have shown that some individuals can pre- dict the outcome of future events with con- sistently better accuracy. This leads to a nat- ural question: what makes some forecasters better than others? In this paper we explore connections between the language people use to describe their predictions and their forecast- ing skill. Datasets from two different fore- casting domains are explored: (1) geopolitical forecasts from Good Judgment Open, an on- line prediction forum and (2) a corpus of com- pany earnings forecasts made by financial an- alysts. We present a number of linguistic met- rics which are computed over text associated with people’s predictions about the future in- cluding: uncertainty, readability, and emotion. By studying linguistic factors associated with predictions, we are able to shed some light on the approach taken by skilled forecasters. Fur- thermore, we demonstrate that it is possible to accurately predict forecasting skill using a model that is based solely on language. This could potentially be useful for identifying ac- curate predictions or potentially skilled fore- casters earlier.1 1 Introduction People often make predictions about the future, for example meteorologists tell us what the weather might look like tomorrow, financial analysts predict which companies will report favorable earnings and intelligence analysts evaluate the likelihood of future geopolitical events. An interesting question is why some individuals are significantly better forecasters (Mellers et al., 2015b)? Previous work has analyzed to what degree vari- ous factors (intelligence, thinking style, knowledge 1We provide our code and dataset descriptions at: https://github.com/viczong/measuring_ forecasting_skill_from_text. of a specific topic, etc.) contribute to a person’s skill. These studies have used surveys or psycho- logical tests to measure dispositional, situational and behavioral variables (Mellers et al., 2015a). Another source of information has been largely overlooked, however: the language forecasters use to justify their predictions. Recent research has demonstrated that it is possible to accurately fore- cast the outcome of future events by aggregating social media users’ predictions and analyzing their veridicality (Swamy et al., 2017), but to our knowl- edge, no prior work has investigated whether it might be possible to measure a forecaster’s ability by analyzing their language. In this paper, we present the first systematic study of the connection between language and fore- casting ability. To do so, we analyze texts writ- ten by top forecasters (ranked by accuracy against ground truth) in two domains: geopolitical fore- casts from an online prediction forum, and com- pany earnings forecasts made by financial analysts. To shed light on the differences in approach em- ployed by skilled and unskilled forecasters, we in- vestigate a variety of linguistic …

Content truncated to 500 words