Think about it! Improving defeasible reasoning by first modeling the question scenario.

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6291–6310 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6291 Think about it! Improving defeasible reasoning by first modeling the question scenario Aman Madaan, Niket Tandon†, Dheeraj Rajagopal, Peter Clark†, Yiming Yang, Eduard Hovy Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA, USA † Allen Institute for Artificial Intelligence, Seattle, WA, USA {dheeraj,amadaan,yiming,hovy}@cs.cmu.edu {nikett, peterc}@allenai.org Abstract Defeasible reasoning is the mode of reasoning where conclusions can be overturned by taking into account new evidence. Existing cognitive science literature on defeasible reasoning sug- gests that a person forms a mental model of the problem scenario before answering questions. Our research goal asks whether neural mod- els can similarly benefit from envisioning the question scenario before answering a defeasi- ble query. Our approach is, given a question, to have a model first create a graph of relevant influences, and then leverage that graph as an additional input when answering the question. Our system, CURIOUS, achieves a new state- of-the-art on three different defeasible reason- ing datasets. This result is significant as it il- lustrates that performance can be improved by guiding a system to “think about” a question and explicitly model the scenario, rather than answering reflexively.1 1 Introduction Defeasible inference is a mode of reasoning where additional information can modify conclusions (Koons, 2017). Here we consider the specific for- mulation and challenge in Rudinger et al. (2020): Given that some premise P plausibly implies a hy- pothesis H, does new information that the situation is S weaken or strengthen the conclusion H? For example, consider the premise “The drinking glass fell” with a possible implication “The glass broke”. New information that “The glass fell on a pillow” here weakens the implication. We borrow ideas from the cognitive science liter- ature that supports defeasible reasoning for humans with an inference graph (Pollock, 2009, 1987). In- ference graphs formulation in (Madaan and Yang, 2021), which we use in this paper, draws connec- tions between the P, H, and S through mediating 1Data and code located at https://github.com/ madaan/thinkaboutit Figure 1: CURIOUS improves defeasible reasoning by modeling the question scenario with an inference graph G adapted from cognitive science literature. The graph is encoded judiciously using our graph encoder h(.), resulting in end task performance improvement. events. This can be seen as a mental model of the question scenario before answering the question (Johnson-Laird, 1983). This paper asks the natu- ral question: can modeling the question scenario with inference graphs help machines in defeasible reasoning? Our approach is as follows. First, given a ques- tion, generate an inference graph describing impor- tant influences between question elements. Then, use that graph as an additional input when answer- ing the defeasible reasoning query. Our proposed system, CURIOUS, comprises a graph generation module and a graph encoding module to use the generated graph for the query (Figure 2). To generate inference graphs, we build upon past work that uses a sequence to sequence approach …

Content truncated to 500 words