APOLLO: A Simple Approach for Adaptive Pretraining of Language Models for Logical Reasoning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6308–6321 July 9-14, 2023 ©2023 Association for Computational Linguistics APOLLO: A Simple Approach for Adaptive Pretraining of Language Models for Logical Reasoning Soumya Sanyal1∗ Yichong Xu2 Shuohang Wang2 Ziyi Yang2 Reid Pryzant2 Wenhao Yu3∗ Chenguang Zhu2 Xiang Ren1 1University of Southern California 2Microsoft Cognitive Service Research 3University of Notre Dame soumyasa@usc.edu Abstract Logical reasoning over text is an important abil- ity that requires understanding the semantics of the text and reasoning through them to arrive at correct inferences. Prior works on pretraining language models to improve the logical rea- soning ability require complex processing of training data (e.g., aligning symbolic knowl- edge to text), yielding task-specific solutions that are not easy to adapt to any general text corpus. In this work, we propose APOLLO, a simple adaptive pretraining approach to im- prove the logical reasoning skills of language models. We select a subset of Wikipedia for adaptive pretraining using a set of logical in- ference keywords as filter words. Further, we propose two self-supervised loss functions for training. First, we modify the masked language modeling loss to mask specific parts-of-speech words that likely require higher-order reason- ing to predict them. Second, we propose a sentence-level classification loss that teaches the model to distinguish between entailment and contradiction types of sentences. The pro- posed pretraining paradigm is both simple and independent of task formats. We demonstrate the effectiveness of APOLLO by comparing it with prior baselines on two logical reasoning datasets. APOLLO performs comparably on ReClor and outperforms baselines on LogiQA. The code base has been made publicly avail- able.1 1 Introduction Logical reasoning is an important ability of hu- mans that helps us in making rational decisions based on known information. It is an important ability for text understanding across various down- stream tasks, e.g., in open-domain question answer- ing (Yang et al., 2018; Zhu et al., 2021), machine ∗ Work done during internship at Microsoft Cognitive Service Research group 1https://github.com/INK-USC/APOLLO Input Sentence: Earth’s temperature would drop down if it were frozen entirely, because it would be more reflective. Random Masking: [MASK] temperature [MASK] drop [MASK] if it were frozen entirely, because it would be [MASK] reflective. Selective Masking: Earth’s temperature would [MASK] down if it were [MASK] entirely, [MASK] it would be [MASK] reflective. Figure 1: Motivation of Selective Masking. In random masking (Devlin et al., 2019), a word is masked at random. Predicting these words often require more of language under- standing than higher-order reasoning (e.g., predicting “would” at the 2nd [MASK] place). In selective masking, a word is masked if its POS tag is from a specific set. These candidate words are marked in the blue box in the input sentence. Filling these words requires more reasoning (e.g., to predict “more” at the 2nd [MASK] place instead of “less”, which is also gram- matically valid, the model needs a better understanding of the semantics of the sentence). reading comprehension (MRC) …

Content truncated to 500 words