RuleBERT: Teaching Soft Rules to Pre-Trained Language Models

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1460–1476 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1460 RULEBERT: Teaching Soft Rules to Pre-Trained Language Models Mohammed Saeed† Naser Ahmadi† Preslav Nakov‡ Paolo Papotti† †Eurecom, France ‡Qatar Computing Research Institute, HBKU, Qatar { FirstName.LastName }@eurecom.fr, pnakov@hbku.edu.qa Abstract While pre-trained language models (PLMs) are the go-to solution to tackle many natural language processing problems, they are still very limited in their ability to capture and to use common-sense knowledge. In fact, even if information is available in the form of approx- imate (soft) logical rules, it is not clear how to transfer it to a PLM in order to improve its per- formance for deductive reasoning tasks. Here, we aim to bridge this gap by teaching PLMs how to reason with soft Horn rules. We in- troduce a classification task where, given facts and soft rules, the PLM should return a pre- diction with a probability for a given hypoth- esis. We release the first dataset for this task, and we propose a revised loss function that en- ables the PLM to learn how to predict precise probabilities for the task. Our evaluation re- sults show that the resulting fine-tuned models achieve very high performance, even on logi- cal rules that were unseen at training. More- over, we demonstrate that logical notions ex- pressed by the rules are transferred to the fine- tuned model, yielding state-of-the-art results on external datasets. 1 Introduction Pre-trained language models (PLMs) based on transformers (Devlin et al., 2019; Liu et al., 2020) are established tools for capturing both linguistic and factual knowledge (Clark et al., 2019b; Rogers et al., 2020). However, even the largest models fail on basic reasoning tasks. If we consider com- mon relations between entities, we see that such models are not aware of negation, inversion (e.g., parent-child), symmetry (e.g., spouse), implica- tion, and composition. While these are obvious to a human, they are challenging to learn from text corpora as they go beyond linguistic and fac- tual knowledge (Ribeiro et al., 2020; Kassner and Schütze, 2020). We claim that such reasoning prim- itives can be transferred to the PLMs by leveraging logical rules, such as those shown in Figure 1. Input facts: Mike is the parent of Anne. Anne lives with Mark. Anne is the child of Laure. Anne lives with Mike. Input rules: (r1, .1) Two persons living together are married. (r2, .7) Persons with a common child are married. (r3, .9) Someone cannot be married to his/her child. (r4, 1) Every person is the parent of his/her child. Test 1: Laure and Mike are married. Answer: True with probability 0.7 [r4, r2] Test 2: Anne and Mark are married. Answer: False with probability 0.9 [r1] Test 3: Anne and Mike are married. Answer: False with probability 0.9 [r1, r3, r4] Figure 1: Examples of hypotheses that require reason- ing using facts and possibly conflicting soft rules (rule id and confidence shown in brackets). While there …

Content truncated to 500 words