Could you give me a hint ? Generating inference graphs for defeasible reasoning

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 5138–5147 August 1–6, 2021. ©2021 Association for Computational Linguistics 5138 Could you give me a hint? Generating inference graphs for defeasible reasoning Aman Madaan ∗, Dheeraj Rajagopal ∗, Niket Tandon ∗ †, Yiming Yang, Eduard Hovy Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA, USA † Allen Institute for Artificial Intelligence, Seattle, WA, USA {dheeraj,amadaan,yiming,hovy}@cs.cmu.edu {nikett}@allenai.org Abstract Defeasible reasoning is a mode of reasoning where conclusions can be overturned by tak- ing into account new evidence. A commonly used method in cognitive science and logic lit- erature is to handcraft argumentation support- ing inference graphs. While humans find in- ference graphs very useful for reasoning, con- structing them at scale is difficult. In this pa- per, we automatically generate such inference graphs through transfer learning from a related NLP task that shares the kind of reasoning that inference graphs support. Through automated metrics and human evaluation, we find that our method generates meaningful graphs for the defeasible inference task. Human accuracy on this task improves by 20% by consulting the generated graphs. Our findings open up excit- ing new research avenues for cases where ma- chine reasoning can help human reasoning.1 1 Introduction Defeasible inference (Rudinger et al., 2020) is a mode of reasoning in which given a premise P (Rob went for a hike), a hypothesis H (Rob saw an elephant, it was pink) may be weakened or overturned in light of new evidence i.e., an up- date U (Rob often has hallucinations). Given the non-monotonic nature of this reasoning, humans find it challenging to master this task (Morgan, 2004). This problem has been widely studied in classical AI through logic (Israel, 1980; McCarthy, 1981), and in cognitive science through argumenta- tive models (Pollock, 1987). A prominent approach is to support defeasible inference through argumen- tations by constructing an inference graph (Pollock, 2009). ∗Equal Contribution 1A dataset of 230,000 influence graphs for each de- feasible query is located at: https://tinyurl.com/ defeasiblegraphs. Despite their prominence (Bentahar et al., 2010), argumentative models are not scalable because an inference graph needs to be handcrafted for every example. Recently, Rudinger et al. (2020) proposed two auxiliary tasks related to defeasible inference: (i) an NLI task to predict whether an update U would weaken or strengthen a hypothesis H, and (ii) a generative task to generate an update U given a premise P and a hypothesis H. However, this only addresses a part of the problem because their inference is still not supported by the line of rea- soning that a human typically uses to solve this task, namely mediators (e.g., hallucinations can be deceptive) and contextualizers (some elephants can have mutated gene which makes them look differ- ent) that are inherently embedded in an inference graph, limiting their utility for humans (figure 1). In this paper, we adopt the concept of an infer- ence graph for defeasible reasoning from cognitive science and provide a computational model to make their generation scalable. Training such …

Content truncated to 500 words