Diagnosing the First-Order Logical Reasoning Ability Through LogicNLI

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3738–3747 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3738 Diagnosing the First-Order Logical Reasoning Ability Through LogicNLI Jidong Tian1,2†, Yitian Li1,2†, Wenqing Chen1,2, Liqiang Xiao1,2, Hao He1,2‡ and Yaohui Jin1,2 1MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University 2State Key Lab of Advanced Optical Communication System and Network, Shanghai Jiao Tong University {frank92, yitian_li, wenqingchen, xiaoliqiang, hehao, jinyh}@sjtu.edu.cn Abstract Recently, language models (LMs) have achieved significant performance on many NLU tasks, which has spurred widespread interest for their possible applications in the scientific and social area. However, LMs have faced much criticism of whether they are truly capable of reasoning in NLU. In this work, we propose a diagnostic method for first-order logic (FOL) reasoning with a new proposed benchmark, LogicNLI. LogicNLI is an NLI-style dataset that effectively disentangles the target FOL reasoning from commonsense inference and can be used to diagnose LMs from four perspectives: accuracy, robustness, generalization, and traceability. Experiments on BERT, RoBERTa, and XLNet, have uncov- ered the weaknesses of these LMs on FOL reasoning, which motivates future exploration to enhance the reasoning ability. 1 Introduction Recently, Transformers-based (Vaswani et al., 2017) language models (LMs), such as BERT (De- vlin et al., 2019) and RoBERTa (Liu et al., 2019), have achieved great success on natural language understanding (NLU). However, there are grow- ing concerns about whether LMs can truly under- stand natural language or not. Tasks with complex reasoning have provided evidence that LMs lack expected reasoning abilities (Liu et al., 2020; Bha- gavatula et al., 2020). Even if neural models can make correct predictions, they tend to make deci- sions through spurious statistical correlations rather than reasoning abilities (Kaushik and Lipton, 2018; Ribeiro et al., 2019; Jiang and Bansal, 2019; Mc- Coy et al., 2019). Therefore, an increasing number of studies have focused on diagnosing specific rea- soning abilities of state-of-the-art LMs (Sugawara et al., 2020; Gontier et al., 2020). † These authors contributed equally. ‡ Corresponding author. First-order logical (FOL) reasoning is one of the most widely used reasoning forms in natural lan- guage (Davis, 2017; Yu et al., 2020), which has a simple paradigm consisting of combinations of seven fundamental logics (FOLs, including con- junction ∧, disjunction ∨, negation ¬, implication →, equation ≡, universal quantifier ∀, and existen- tial quantifier ∃) with simple propositions (Davis, 2017). Nevertheless, whether LMs can truly make FOL reasoning is still inconclusive in NLP (Hahn et al., 2021; Clark et al., 2020). As a result, we propose a systematic diagnostic method for FOL reasoning by proposing a novel benchmark, named Logical Natural Language Inference (LogicNLI). The proposed benchmark follows three principles: 1) It includes abundant logical expressions covering all seven FOLs and their commonly used combinations in texts; 2) The instances of the benchmark conform to natural lan- guage; 3) It introduces as little commonsense as possible to prevent the targeting FOL reasoning and commonsense inference from being entangled with …

Content truncated to 500 words