MERIt: Meta-Path Guided Contrastive Learning for Logical Reasoning

Findings of the Association for Computational Linguistics: ACL 2022, pages 3496 - 3509 May 22-27, 2022 c⃝2022 Association for Computational Linguistics MERIt: Meta-Path Guided Contrastive Learning for Logical Reasoning Fangkai Jiao†, Yangyang Guo§∗, Xuemeng Song†, Liqiang Nie†∗ †School of Computer Science and Technology, Shandong University, Qingdao, China §School of Computing, National University of Singapore jiaofangkai@hotmail.com {guoyang.eric, sxmustc, nieliqiang}@gmail.com Abstract Logical reasoning is of vital importance to nat- ural language understanding. Previous stud- ies either employ graph-based models to in- corporate prior knowledge about logical rela- tions, or introduce symbolic logic into neu- ral models through data augmentation. These methods, however, heavily depend on anno- tated training data, and thus suffer from over- fitting and poor generalization problems due to the dataset sparsity. To address these two problems, in this paper, we propose MERIt, a MEta-path guided contrastive learning method for logical ReasonIng of text, to perform self- supervised pre-training on abundant unlabeled text data. Two novel strategies serve as indis- pensable components of our method. In partic- ular, a strategy based on meta-path is devised to discover the logical structure in natural texts, followed by a counterfactual data augmenta- tion strategy to eliminate the information short- cut induced by pre-training. The experimental results on two challenging logical reasoning benchmarks, i.e., ReClor and LogiQA, demon- strate that our method outperforms the SOTA baselines with significant improvements.1 1 Introduction Logical reasoning has long been recognized as one key critical thinking ability of human being. Un- til very recently, some pioneer researchers have crystallized this for the NLP community, and built several public challenging benchmarks, such as ReColor (Yu et al., 2020) and LogiQA (Liu et al., 2020). Logical reasoning2 requires to correctly infer the semantic relations with respect to the constituents among different sentences. A typi- cal formulation of logical reasoning is illustrated *Corresponding author: Yangyang Guo and Liqiang Nie. 1Our code and pre-trained models are available at https: //github.com/SparkJiao/MERIt. 2We refer the term logical reasoning to the task itself in the remaining of this paper. Figure 1: An instance of logical reasoning from the Re- Clor dataset. To infer the right answer, we should un- cover the underlying logical structure, as shown in the bottom. (x) represents the logical variable (e.g., entity or phrase) and rj denotes the relation (e.g., predicate) between two logical variables. ¯rj is the passive relation of rj. in Figure 1, namely, a real-world examination in- stance from ReClor. As can be seen, to find the correct answer for the given question, one needs to extract the logical structures residing in a pair of each option and the whole context, and justify its reasonableness. As a matter of fact, logical reasoning is still at its initial stage, thence, existing studies are somewhat rare in literature. Some efforts have been devoted to designing specific model architectures or inte- grating symbolic logic as the hints attached to the potential logical structure. For instance, Huang et al. (2021) and Ouyang et al. (2021) first con- structed a graph of different …

Content truncated to 500 words