Explainable Multi-hop Verbal Reasoning Through Internal Monologue

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1225–1250 June 6–11, 2021. ©2021 Association for Computational Linguistics 1225 Explainable Multi-hop Verbal Reasoning Through Internal Monologue Zhengzhong Liang1, Steven Bethard2 and Mihai Surdeanu1 1 Computer Science Department, The University of Arizona, Tucson, AZ 2School of Information, The University of Arizona, Tucson, AZ {zhengzhongliang, bethard, msurdeanu}@email.arizona.edu Abstract Many state-of-the-art (SOTA) language mod- els have achieved high accuracy on several multi-hop reasoning problems. However, these approaches tend to not be interpretable because they do not make the intermediate reasoning steps explicit. Moreover, models trained on simpler tasks tend to fail when di- rectly tested on more complex problems. We propose the Explainable multi-hop Verbal Rea- soner (EVR) to solve these limitations by (a) decomposing multi-hop reasoning problems into several simple ones, and (b) using natu- ral language to guide the intermediate reason- ing hops. We implement EVR by extending the classic reasoning paradigm General Prob- lem Solver (GPS) with a SOTA generative lan- guage model to generate subgoals and perform inference in natural language at each reason- ing step. Evaluation of EVR on Clark et al. (2020)’s synthetic question answering (QA) dataset shows that EVR achieves SOTA per- formance while being able to generate all rea- soning steps in natural language. Furthermore, EVR generalizes better than other strong meth- ods when trained on simpler tasks or less train- ing data (up to 35.7% and 7.7% absolute im- provement respectively).1 1 Introduction Large pretrained language models such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) have been successfully used in multi-hop reasoning problems (Banerjee et al., 2020; Asai et al., 2019; Yadav et al., 2019). Usually, these pre- trained language models solve multi-hop reasoning problems in a discriminative end-to-end manner: these models take the question and all the relevant evidence as the input, and produce the final answer to the question. This raises two problems. First, this direction lacks interpretability, i.e., it is hard to 1The code is available at: https://github. com/clulab/releases/tree/master/ naacl2021-evr (Input Facts:) Alan is blue. Alan is rough. Alan is young. Bob is big. Bob is round. Charlie is big. Charlie is blue. Charlie is green. Dave is green. Dave is rough. (Input Rules:) Big people are rough. If someone is young and round then they are kind. If someone is round and big then they are blue. All rough people are green. Q1: Bob is green. True/false? [Answer: T] Q2: Bob is kind. True/false? [Answer: F] Figure 1: An example taken from (Clark et al., 2020). The context includes two types of statements: facts and rules. Multiple facts and rules are usually needed to answer the questions. For example, to prove “Bob is green,” the model needs to construct the reasoning chain “All rough people are green ← Big people are rough ← Bob is big”. know which individual reasoning steps are taken in each iteration and why. Second, the trained models usually suffer …

Content truncated to 500 words