MetaLogic: Logical Reasoning Explanations with Fine-Grained Structure

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4698 - 4724 December 7-11, 2022 ©2022 Association for Computational Linguistics MetaLogic: Logical Reasoning Explanations with Fine-Grained Structure Yinya Huang1,2∗ Hongming Zhang2† Ruixin Hong3 Xiaodan Liang1,4† Changshui Zhang3 Dong Yu2 1Shenzhen Campus of Sun Yat-sen University 2Tencent AI Lab, Seattle 3Tsinghua University 4Pengcheng Laboratory yinya.huang@hotmail.com, {hongmzhang, dyu}@global.tencent.com, hrx20@mails.tsinghua.edu.cn, zcs@mail.tsinghua.edu.cn, xdliang328@gmail.com Abstract In this paper, we propose a comprehensive benchmark to investigate models’ logical rea- soning capabilities in complex real-life scenar- ios. Current explanation datasets often employ synthetic data with simple reasoning structures. Therefore, it cannot express more complex rea- soning processes, such as the rebuttal to a rea- soning step and the degree of certainty of the evidence. To this end, we propose a comprehen- sive logical reasoning explanation form. Based on the multi-hop chain of reasoning, the expla- nation form includes three main components: (1) The condition of rebuttal that the reasoning node can be challenged; (2) Logical formulae that uncover the internal texture of reasoning nodes; (3) Reasoning strength indicated by de- grees of certainty. The fine-grained structure conforms to the real logical reasoning scenario, better fitting the human cognitive process but, simultaneously, is more challenging for the cur- rent models. We evaluate the current best mod- els’ performance on this new explanation form. The experimental results show that generating reasoning graphs remains a challenging task for current models, even with the help of giant pre-trained language models. 1 Introduction Being able to generate reasonable explanations is a crucial capability for a reliable reasoning sys- tem. Most current works try to ask models to gen- erate reasoning chains as profound explanations. From simple rationales (DeYoung et al., 2020) to more complex multi-step explanations (Inoue et al., 2020; Jhamtani and Clark, 2020; Saha et al., 2021) and deductive chains of reasoning (Clark et al., 2020; Tafjord et al., 2021; Dalvi et al., 2021), pre- vious works attempt to encompass comprehensive information. However, the current explanation de- sign still has limitations for logical reasoning texts ∗ This work was done when Y. Huang was an intern at Tencent AI Lab. †X. Liang and H. Zhang are the co-corresponding authors. Sent1 Certainty: contingent Node Formula: N/A Sent4 Certainty: necessary Node Formula: □(v𝟐 → □v𝟑) Condition of rebuttal Sent3 Certainty: necessary Node Formula: □(v𝟑⋀v𝟐) Sent2 Certainty: contingent Node Formula: N/A Sent5 Certainty: unnecessary Node Formula: ¬□(¬□v𝟑→ v𝟏) Logic Metagraph sent1: v1: doctor : v2: recent pharmaceutical advances will lead the way in weight loss . sent2: v1: prior to these advancements , v2: obesity - related deaths outnumbered all other causes of death by a wide margin . sent3: v1: the new drugs will v2: curb appetite and v3: increase metabolism . sent4: v1: thanks to v2: these advancements , v3: obesity will dramatically decline in the near future . sent5: v1: most people will not be able to afford these prescriptions v2: since v3: the majority of health care plans will not cover the new drugs . Passage …

Content truncated to 500 words