Teach the Rules, Provide the Facts: Targeted Relational-knowledge Enhancement for Textual Inference

Proceedings of the 10th Conference on Lexical and Computational Semantics, pages 89–98 August 5–6, 2021, Bangkok, Thailand (online) ©2021 Association for Computational Linguistics 89 Teach the Rules, Provide the Facts: Targeted Relational-knowledge Enhancement for Textual Inference Ohad Rozen1 Shmuel Amar1 Vered Shwartz2,3 Ido Dagan1 1Computer Science Department, Bar-Ilan University, Ramat-Gan, Israel 2Allen Institute for Artificial Intelligence 3Paul G. Allen School of Computer Science & Engineering, University of Washington {ohadrozen,shmulikamar}@gmail.com, vereds@allenai.org, dagan@cs.biu.ac.il Abstract We present InferBert, a method to enhance transformer-based inference models with rele- vant relational knowledge. Our approach fa- cilitates learning generic inference patterns re- quiring relational knowledge (e.g. inferences related to hypernymy) during training, while injecting on-demand the relevant relational facts (e.g. pangolin is an animal) at test time. We apply InferBERT to the NLI task over a di- verse set of inference types (hypernymy, loca- tion, color, and country of origin), for which we collected challenge datasets. In this set- ting, InferBert succeeds to learn general infer- ence patterns, from a relatively small number of training instances, while not hurting perfor- mance on the original NLI data and substan- tially outperforming prior knowledge enhance- ment models on the challenge data. It further applies its inferences successfully at test time to previously unobserved entities. InferBert is computationally more efficient than most prior methods, in terms of number of parameters, memory consumption and training time. 1 Introduction Transformer-based pre-trained language models (LMs), such as BERT (Devlin et al., 2019) and GPT (Radford et al., 2018) have recently achieved human-level performance on standard natural lan- guage inference (NLI) benchmarks (Wang et al., 2019). However, the performance on this complex task is achieved in part thanks to large training sets that facilitate learning of dataset-specific biases and correlations, and thanks to the similar distributions between the training and test sets, that rewards such models (Poliak et al., 2018; Gururangan et al., 2018). This contrasts with humans, who can learn a generalized solution from fewer examples (Linzen, 2020). Indeed, NLI models often fail on exam- ples involving various linguistic phenomena such as co-hyponymy (Glockner et al., 2018) and nega- tion (Naik et al., 2018), which they are expected to acquire indirectly from the NLI training set. Prior work proposed to provide (“inoculate”) NLI models with a small number of phenomenon- specific training examples in order to teach the model to address them (Liu et al., 2019a). However, Rozen et al. (2019) showed that when the distribu- tions of the training and test sets differ with respect to syntactic and lexical properties, the performance of such inoculated models drops, concluding that they do not learn a generalized notion of the phe- nomenon. In this paper we are motivated by the following question: how can we facilitate learning of generalized inference patterns, with respect to a given linguistic phenomenon, from a relatively small number of examples? Ideally, we would like an NLI model to learn inference patterns detached from their original con- text, and to be able to apply them in new contexts involving …

Content truncated to 500 words