RLET: A Reinforcement Learning Based Approach for Explainable QA with Entailment Trees

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7177 - 7189 December 7-11, 2022 ©2022 Association for Computational Linguistics RLET: A Reinforcement Learning Based Approach for Explainable QA with Entailment Trees Tengxiao Liu1∗, Qipeng Guo2, Xiangkun Hu2, Yue Zhang3†, Xipeng Qiu1†, Zheng Zhang2 1School of Computer Science, Fudan University 2Amazon AWS AI, 3School of Engineering, Westlake University txliu21@m.fudan.edu.cn, {gqipeng, xiangkhu, zhaz}@amazon.com xpqiu@fudan.edu.cn, zhangyue@westlake.edu.cn Abstract Interpreting the reasoning process from ques- tions to answers poses a challenge in approach- ing explainable QA. A recently proposed struc- tured reasoning format, entailment tree, man- ages to offer explicit logical deductions with entailment steps in a tree structure. To generate entailment trees, prior single pass sequence-to- sequence models lack visible internal decision probability, while stepwise approaches are su- pervised with extracted single step data and cannot model the tree as a whole. In this work, we propose RLET, a Reinforcement Learning based Entailment Tree generation framework, which is trained utilising the cumulative signals across the whole tree. RLET iteratively per- forms single step reasoning with sentence selec- tion and deduction generation modules, from which the training signal is accumulated across the tree with elaborately designed aligned re- ward function that is consistent with the evalu- ation. To the best of our knowledge, we are the first to introduce RL into the entailment tree generation task. Experiments on three settings of the EntailmentBank dataset demonstrate the strength of using RL framework. 1 Introduction Reasoning over explicitly given knowledge and generating detailed deduction steps are important challenges towards the goal of automated reason- ing in AI community (McCarthy, 1960; Wos, 1985; Mercier and Sperber, 2011). Interpreting the rea- soning process in QA systems can provide human- understandable inspections into the logical deduc- tions and help ensure the soundness and reliability of the models (Wiegreffe and Marasovi´c, 2021). As shown in Figure 1, given the question “What keeps Mars in orbit around the Sun?”, a reasoning process can explain why the answer “Gravity” is predicted, thereby strengthening the trustworthiness. ∗Work done during internship at AWS Shanghai AI Lab. †Corresponding authors. 𝑥! + 𝑖!: gravity causes the planets in the 𝑖!: solar system to orbitthe sun 𝑖": mars is a planetin the solar system Q: What keeps Mars in orbit around the Sun? A: Gravity Hypothesis ℎ: gravity causes mars to orbit around the sun 𝑥!: gravity causes orbits 𝑥": mars is a kind of planet 𝑥#: planets in the solar system orbit the sun 𝑥$: mars is located in the solar system 𝑥# 𝑥" + 𝑥$ 𝑖! + 𝑖" ℎ 𝑥! 𝑥# 𝑥" 𝑥$ 𝑖! 𝑖" ℎ Entailment Tree Reasoning Steps Facts Figure 1: The entailment tree generation task. Given QA pair, hypothesis (green boxes) and fact sentences (grey boxes), the model needs to generate tree-structured rea- soning chain along with natural language intermediate conclusions (blue boxes). Recent work in the field of explainable QA in- cludes extracting keywords or sentences that indi- cate the answers (Yang et al., 2018), constructing multi-hop …

Content truncated to 500 words