Probing neural language models for understanding of words of estimative probability

Proceedings of the The 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 469–476 July 13-14, 2023 ©2023 Association for Computational Linguistics Probing neural language models for understanding of words of estimative probability Damien Sileo1 and Marie-Francine Moens2 1Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 - CRIStAL, F-59000 Lille, France 2Department of Computer Science, KU Leuven, Belgium damien.sileo@inria.fr Abstract Words of Estimative Probability (WEP) are phrases used to express the plausibility of a statement. Examples include terms like proba- bly, maybe, likely, doubt, unlikely, and impossi- ble. Surveys have shown that human evaluators tend to agree when assigning numerical prob- ability levels to these WEPs. For instance, the term highly likely equates to a median proba- bility of 0.90±0.08 according to a survey by Fagen-Ulmschneider (2015). In this study, our focus is to gauge the competency of neural lan- guage processing models in accurately captur- ing the consensual probability level associated with each WEP. Our first approach is utilizing the UNLI dataset (Chen et al., 2020), which links premises and hypotheses with their per- ceived joint probability p. From this, we craft prompts in the form: "[PREMISE]. [WEP], [HYPOTHESIS]." This allows us to evaluate whether language models can predict if the consensual probability level of a WEP aligns closely with p. In our second approach, we develop a dataset based on WEP-focused prob- abilistic reasoning to assess if language mod- els can logically process WEP compositions. For example, given the prompt "[EVENTA] is likely. [EVENTB] is impossible.", a well- functioning language model should not con- clude that [EVENTA&B] is likely. Through our study, we observe that both tasks present challenges to out-of-the-box English language models. However, we also demonstrate that fine-tuning these models can lead to significant and transferable improvements. 1 Introduction Expression of uncertainty is an important part of communication. Formal statistics are the rigorous way to quantify uncertainty but do not fit all com- munication styles. Words of estimative probability (WEP) such as maybe and believe are adverbs or verbs that are informal alternatives. Kent (1964) noted the importance of clarifying WEP meaning for intelligence analysis in the Central Intelligence Agency, and provided guidelines for mapping WEP to numerical probabilities. Several studies then measured the human perceptions of probability words and discovered some agreement with Kent (1964)’s guidelines. In this work, we use the scale derived from a survey (Fagen-Ulmschneider, 2015), which is the largest and most recent WEP percep- tion survey available. 123 participants were asked to label WEP with numerical probabilities. We use the median of the participant answers to assign a consensual value to each WEP. Associated prob- abilities for the 19 WEP we use are available in Appendix A, table 2. Here, we assess whether neural language mod- els learn the consensual probability judgment of WEP from language modeling pretraining. We develop datasets and a methodology to probe neu- ral language model understanding of WEP. The first dataset leverages previously annotated proba- bility scores between a premise and a hypothesis, in …

Content truncated to 500 words