How Fast can BERT Learn Simple Natural Language Inference?

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 626–633 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 626 Abstract This paper empirically studies whether BERT can really learn to conduct natural language inference (NLI) without utilizing hidden dataset bias; and how efficiently it can learn if it could. This is done via creating a simple entailment judgment case which involves only binary predicates in plain English. The results show that the learning process of BERT is very slow. However, the efficiency of learning can be greatly improved (data reduction by a factor of 1,500) if task-related features are added. This suggests that domain knowledge greatly helps when conducting NLI with neural networks. 1 Introduction Entailment judgment (Dagan et al., 2006; Marelli et al., 2014a) is a common test for natural language inference (NLI) (Camburu et al., 2018; Conneau et al., 2018) as it possesses the simplest form in related tasks such as question and answering (Bowman and Zhu, 2019). Also, SNLI dataset (Bowman et al., 2015) is frequently adopted for NLI evaluation because it is the first corpus to show the power of neural networks for the task that specifically targets NLI. Recently, BERT (Devlin et al., 2019), XLNet (Yang et al., 2019), and RoBERTa (Liu et al., 2019) have all shown excellent performances on SNLI, surpassing even human performance (Gong et al., 2017). However, Tsuchiya (2018) and Gururangan et al. (2018) show that SNLI contains hidden bias. Also, deep neural networks (DNNs) have been shown to predominantly capture the statistical irregularities that go unnoticed by humans (Poliak et al., 2018); this surprisingly yields 69% accuracy on SNLI without being provided the associated premise (i.e., the required supporting evidence). Furthermore, some studies (Naik et al., 2018; McCoy et al., 2019; Jiang and Marneffe, 2019) have shown that BERT mainly conducts NLI with surface clues/patterns, but not those clues actually adopted by humans. To the best of our knowledge, none of previous studies carefully removes the dataset bias, and then investigates how efficiently that BERT could learn NLI if the surface clues are completely removed. In this paper, we empirically study whether BERT is capable of learning NLI without surface clues/bias appearing in the dataset; and if it is, whether it can learn NLI efficiently. We carefully created absolutely unbiased datasets, in which the premises and hypotheses simply describe the relative position of two objects in plain English. That is, given a premise “John is on the left side of Mary” (abbreviated as a predicate “left(John, Mary)” from now on), the hypotheses predicates “left(John, Mary)”, “left(Mary, John)” and “left(John, Helen)” should be labeled as entailment, contradictory and neutral respectively. Our experiment results show BERT is very slow in learning this simple NLI. We then further study if the learning efficiency can be improved with domain knowledge (Gülçehre and Bengio, 2016). Inspired by Chen et al. (2017), we think whether two entities are exactly the same is quite crucial in making the …

Content truncated to 500 words