He Thinks He Knows Better than the Doctors: BERT for Event Factuality Fails on Pragmatics

He Thinks He Knows Better than the Doctors: BERT for Event Factuality Fails on Pragmatics Nanjiang Jiang Department of Linguistics The Ohio State University, USA jiang.1879@osu.edu Marie-Catherine de Marneffe Department of Linguistics The Ohio State University, USA demarneffe.1@osu.edu Abstract We investigate how well BERT performs on predicting factuality in several existing English datasets, encompassing various linguistic con- structions. Although BERT obtains a strong performance on most datasets, it does so by exploiting common surface patterns that cor- relate with certain factuality labels, and it fails on instances where pragmatic reasoning is nec- essary. Contrary to what the high performance suggests, we are still far from having a robust system for factuality prediction. 1 Introduction Predicting event factuality1 is the task of iden- tifying to what extent an event mentioned in a sentence is presented by the author as factual. It is a complex semantic and pragmatic phenomenon: in John thinks he knows better than the doctors, we infer that John probably doesn’t know better than the doctors. Event factuality inference is prevalent in human communication and matters for tasks that depend on natural language understanding, such as information extraction. For instance, in the FactBank example (Saur´ı and Pustejovsky, 2009) in Table 1, an information extraction sys- tem should extract people are stranded without food but not helicopters located people stranded without food. The current state-of-the-art model for factual- ity prediction on English is the work of Pouran Ben Veyseh et al. (2019), obtaining the best per- formance on four factuality datasets: FactBank, MEANTIME (Minard et al., 2016), UW (Lee et al., 2015), and UDS-IH2 (Rudinger et al., 2018). Traditionally, event factuality is thought to be triggered by fixed properties of lexical items. The Rule-based model of Stanovsky et al. (2017) took such an approach: They used lexical rules and 1The terms veridicality and speaker commitment refer to the same underlying linguistic phenomenon. dependency trees to determine whether an event in a sentence is factual, based on the properties of the lexical items that embed the event in ques- tion. Rudinger et al. (2018) proposed the first end-to-end model for factuality with LSTMs. Pouran Ben Veyseh et al. (2019) used BERT rep- resentations with a graph convolutional network and obtained a large improvement over Rudinger et al. (2018) and over Stanovsky et al.’s (2017) Rule-based model (except for one metric on the UW dataset). However, it is not clear what these end-to-end models learn and what features are encoded in their representations. In particular, they do not seem capable of generalizing to events embedded under certain linguistic constructions. White et al. (2018) showed that the Rudinger et al. (2018) models exhibit systematic errors on MegaVeridi- cality, which contains factuality inferences purely triggered by the semantics of clause-embedding verbs in specific syntactic contexts. Jiang and de Marneffe (2019a) showedthat Stanovsky et al.’s and Rudinger et al.’s models fail to perform well on the CommitmentBank (de Marneffe et al., 2019), which contains events under clause- embedding verbs in an entailment-canceling envi- ronment (negation, …

Content truncated to 500 words