Evaluating BERT for natural language inference: A case study on the CommitmentBank

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 6086–6091, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 6086 Evaluating BERT for natural language inference: A case study on the CommitmentBank Nanjiang Jiang and Marie-Catherine de Marneffe Department of Linguistics The Ohio State University {jiang.1879, demarneffe.1}@osu.edu Abstract Natural language inference (NLI) datasets (e.g., MultiNLI) were collected by solicit- ing hypotheses for a given premise from an- notators. Such data collection led to an- notation artifacts: systems can identify the premise-hypothesis relationship without ob- serving the premise (e.g., negation in hy- pothesis being indicative of contradiction). We address this problem by recasting the CommitmentBank for NLI, which contains items involving reasoning over the extent to which a speaker is committed to complements of clause-embedding verbs under entailment- canceling environments (conditional, nega- tion, modal and question). Instead of being constructed to stand in certain relationships with the premise, hypotheses in the recast CommitmentBank are the complements of the clause-embedding verb in each premise, lead- ing to no annotation artifacts in the hypoth- esis. A state-of-the-art BERT-based model performs well on the CommitmentBank with 85% F1. However analysis of model behavior shows that the BERT models still do not cap- ture the full complexity of pragmatic reason- ing, nor encode some of the linguistic general- izations, highlighting room for improvement. 1 Introduction Natural language inference (NLI), the task of identifying whether a hypothesis can be in- ferred from, contradicted by, or not related to a premise, has become one of the standard bench- mark tasks for natural language understanding. NLI datasets, such as SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2018), are typi- cally built by asking annotators to compose sen- tences based on premises extracted from cor- pora, so that the composed sentences stand in en- tailment/contradiction/neutral relationship to the premise. The hypotheses collected this way have Premise: A: Boy that’s scary, isn’t it. B: Oh, can you imag- ine, because it happens in the middle of the night, so you know, these parents didn’t know the kid was gone until the kid is knocking on the door screaming, let me in. Hypothesis: the kid was gone. Entailment (1.92) Premise: “Clever”. Klug means “clever”. Would you say that Abie was clever? Hypothesis: Abie was clever. Neutral (0) Premise: GM confirmed it received U.S. antitrust clearance to boost its holding. Sansui Electric agreed to sell a 51% stake to Polly Peck of Britain for $110 million. Still, analysts said the accord doesn’t suggest Japan is opening up to more foreign takeovers. Hypothesis: Japan is opening up to more foreign takeovers. Contradiction (-1.2) Table 1: Examples from the CommitmentBank, with NLI class and original annotation mean. been found to contain annotation artifacts – clues allowing systems to identify the relationship be- tween a premise and a hypothesis without observ- ing the premise. For instance, Gururangan et al. (2018) found that in SNLI and MultiNLI, …

Content truncated to 500 words