DiscoSense: Commonsense Reasoning with Discourse Connectives

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10295 - 10310 December 7-11, 2022 ©2022 Association for Computational Linguistics DISCOSENSE: Commonsense Reasoning with Discourse Connectives Prajjwal Bhargava and Vincent Ng Human Language Technology Research Institute University of Texas at Dallas Richardson, TX 75083-0688 prajjwalin@protonmail.com, vince@hlt.utdallas.edu Abstract We present DISCOSENSE, a benchmark for commonsense reasoning via understanding a wide variety of discourse connectives. We gen- erate compelling distractors in DISCOSENSE using Conditional Adversarial Filtering, an ex- tension of Adversarial Filtering that employs conditional generation. We show that state-of- the-art pre-trained language models struggle to perform well on DISCOSENSE, which makes this dataset ideal for evaluating next-generation commonsense reasoning systems. 1 Introduction Much of the recent work in commonsense reason- ing has focused on evaluating a pre-trained lan- guage model’s (LM) ability to predict the most plausible ending/option given a context. Even after devising bias reduction techniques (Zellers et al., 2019b; Bras et al., 2020) to mitigate the effects of annotation artifacts and make the task difficult, state-of-the-art LMs have managed to achieve or even surpass human performance on numerous commonsense downstream tasks (Zellers et al., 2019b; Sakaguchi et al., 2020; Bhagavatula et al., 2020). Nevertheless, these LMs are still very far from being able to perform commonsense reason- ing as well as humans. Hence, the fact that they have begun to ace existing benchmarks implies that time is ripe to design a new challenging benchmark that can reliably target their limitations. Motivated by this observation, we present DIS- COSENSE, a benchmark for performing common- sense reasoning through understanding a wide va- riety of discourse connectives. Figure 1 shows an example taken from DISCOSENSE. As can be seen, an example is composed of a context (e.g., “Our waitress was very nice, but she kept on forgetting my stuff.”) and a discourse connective (e.g., “For example”), and the goal is to choose the most plau- sible ending out of four options. If we ignore the discourse connective, then all four options may Our waitress was very nice, but she kept on forgetting my stuff. For example a) When I ordered the garlic shrimp, she remembered to add my requested garlic butter. b) She took forever to bring me my beer and fries. c) When I told her I wanted to use the free breakfast that was available she was not pleased. d) For some customers, this is fine. Figure 1: Example on commonsense reasoning with discourse connectives. The correct (i.e., most plausible) option is boldfaced. seem plausible because we do not know what the writer’s intent is. Once we consider both the con- text and the discourse connective, then it is clear that only option b) is plausible. The reason is that “For example” signals an EXEMPLIFICATION rela- tion between its arguments, and what follows the discourse connective is expected to be an example of the waitress keeping on forgetting the writer’s stuff. Using commonsense knowledge, we know that (1) “my beer and fries” is an example of …

Content truncated to 500 words