Modeling Event Plausibility with Consistent Conceptual Abstraction

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1732–1743 June 6–11, 2021. ©2021 Association for Computational Linguistics 1732 Modeling Event Plausibility with Consistent Conceptual Abstraction Ian Porada1, Kaheer Suleman2, Adam Trischler2, and Jackie Chi Kit Cheung1 1Mila, McGill University {ian.porada@mail, jcheung@cs}.mcgill.ca 2Microsoft Research Montréal {kasulema, adam.trischler}@microsoft.com Abstract Understanding natural language requires com- mon sense, one aspect of which is the ability to discern the plausibility of events. While distributional models—most recently pre-trained, Transformer language models— have demonstrated improvements in modeling event plausibility, their performance still falls short of humans’. In this work, we show that Transformer-based plausibility models are markedly inconsistent across the conceptual classes of a lexical hierarchy, inferring that “a person breathing” is plausible while “a den- tist breathing” is not, for example. We find this inconsistency persists even when models are softly injected with lexical knowledge, and we present a simple post-hoc method of forc- ing model consistency that improves correla- tion with human plausibility judgements. 1 Introduction Of the following events, a human reader can easily discern that (1) and (2) are semantically plausible, while (3) is nonsensical. (1) The person breathes the air. (2) The dentist breathes the helium. (3) The thought breathes the car. This ability is required for understanding nat- ural language: specifically, modeling selectional preference—the semantic plausibility of predicate- argument structures—is known to be implicit in discriminative tasks such as coreference resolution (Hobbs, 1978; Dagan and Itai, 1990; Zhang et al., 2019b), word sense disambiguation (Resnik, 1997; McCarthy and Carroll, 2003), textual entailment (Zanzotto et al., 2006; Pantel et al., 2007), and se- mantic role labeling (Gildea and Jurafsky, 2002; Zapirain et al., 2013). More broadly, modeling semantic plausibility is a necessary component of generative inferences 0.43 0.53 0.53 0.06 0.98 0.42 clothing shirt person worker chef A chef knits clothing. : Very plausible! A worker knits a shirt. : Implausible! Is it plausible an [X] knits a [Y]? Figure 1: Elements in the matrix are the relative plausi- bility score for the event “an [X] knits a [Y]” as output by a RoBERTa model fine-tuned to model plausibility. [X] and [Y] correspond to the label of the row and col- umn, respectively. Model scores are inconsistent with respect to the two events shown on the right. such as conditional commonsense inference (Gor- don et al., 2011; Zhang et al., 2017), abductive commonsense reasoning (Bhagavatula et al., 2020), and commonsense knowledge acquisition (Zhang et al., 2020a; Hwang et al., 2020). Learning to model semantic plausibility is a dif- ficult problem for several reasons. First, language is sparse, so most events will not be attested even in a large corpus. Second, plausibility relates to likelihood in the world, which is distinct from the likelihood of an event occurring in language. Third, plausibility reflects human intuition, and thus mod- eling plausibility at its extreme requires “the entire representational arsenal that people use in under- standing language, ranging from social mores to naive …

Content truncated to 500 words