FaiRR: Faithful and Robust Deductive Reasoning over Natural Language

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1075 - 1093 May 22-27, 2022 c⃝2022 Association for Computational Linguistics FaiRR: Faithful and Robust Deductive Reasoning over Natural Language Soumya Sanyal1 Harman Singh2 Xiang Ren1 1University of Southern California 2Indian Institute of Technology, Delhi {soumyasa, xiangren}@usc.edu, harmansingh.iitd@gmail.com Abstract Transformers have been shown to be able to perform deductive reasoning on a logical rule- base containing rules and statements written in natural language. Recent works show that such models can also produce the reasoning steps (i.e., the proof graph) that emulate the model’s logical reasoning process. Currently, these black-box models generate both the proof graph and intermediate inferences within the same model and thus may be unfaithful. In this work, we frame the deductive logical rea- soning task by defining three modular compo- nents: rule selection, fact selection, and knowl- edge composition. The rule and fact selection steps select the candidate rule and facts to be used and then the knowledge composition com- bines them to generate new inferences. This ensures model faithfulness by assured causal relation from the proof step to the inference reasoning. To test our framework, we pro- pose FAIRR (Faithful and Robust Reasoner) where the above three components are indepen- dently modeled by transformers. We observe that FAIRR is robust to novel language pertur- bations, and is faster at inference than previ- ous works on existing reasoning datasets. Ad- ditionally, in contrast to black-box generative models, the errors made by FAIRR are more interpretable due to the modular approach. 1 1 Introduction The field of AI has long pursued the goal of build- ing systems that can automatically reason over some given explicit knowledge to generate con- clusions and provide the reasoning steps involved in the process (McCarthy, 1959; Newell and Si- mon, 1956). Recently, Clark et al. (2020) proposed a modern version of this problem, where the for- mal representation of knowledge is replaced by natural language statements in English. Further, 1The source code of FAIRR has been made available at https://github.com/INK-USC/FaiRR. fact1: Charlie is blue. fact2: Charlie is round. fact3: Erin is kind. fact4: Dave is round. rule1: If someone is blue then they are kind. rule2: Round, kind people are white. statement: Charlie is white. conc1: Charlie is kind. conc2: Charlie is white. fact1 rule1 fact2 conc1 rule2 conc2 Figure 1: Example of a theory, a statement, and a valid proof graph - An instance contains multiple facts and rules in blue and yellow respectively, followed by a statement in red. The proof graph describes the reason- ing steps required to generate the statement. they proposed a transformer-based model (Vaswani et al., 2017) RuleTaker, that can predict if a candi- date statement is entailed by the natural language statements, by emulating deductive reasoning. As shown in Figure 1, in this deductive reasoning task, facts and rules from the rulebase are combined iter- atively to generate intermediate inferences which eventually entails the statement. Note that the rea- …

Content truncated to 500 words