Re-Examining FactBank: Predicting the Author’s Presentation of Factuality

Proceedings of the 29th International Conference on Computational Linguistics, pages 786–796 October 12–17, 2022. 786 Re-Examining FactBank: Predicting the Author’s Presentation of Factuality John Murzaku⋆3, Peter Zeng⋆, Magdalena Markowska♣3, Owen Rambow♣3 ⋆ Department of Computer Science ♣ Department of Linguistics 3 Institute for Advanced Computational Science Stony Brook University, Stony Brook, NY, USA Corresponding Author: jmurzaku@cs.stonybrook.edu Abstract We present a corrected version of a previous projection of the FactBank data set. Previously published results on FactBank are no longer valid. We perform experiments on FactBank using multiple training paradigms, data smooth- ing techniques, and polarity classifiers. We ar- gue that f-measure is an important alternative evaluation metric for factuality. We provide new state-of-the-art results for four corpora in- cluding FactBank. We perform an error anal- ysis on Factbank combined with two similar corpora. 1 Introduction The term factuality1 refers to an author’s presenta- tion of a proposition (who-did-what-to-whom) as a fact, i.e., she is committed to the truth of the propo- sition. A lot of language use introduces proposi- tions that are not presented as facts but as only pos- sibly true, as a wish or as a hypothesis, or as some- one else’s belief. If we want to understand what an author is communicating, we need to distinguish these cases. Over the last 15 years, this question has received a lot of attention. Multiple corpora have been created, and these corpora have been used to explore machine learning architectures for factuality prediction. The machine learning studies often report results on all corpora, but these studies do not examine what the machine learning architec- ture can learn, nor how and why combining corpora can help. A notable exception is Jiang and de Marn- effe (2021), who carefully analyze what exactly is learned from the CB corpus. In this paper, our goal is to determine how to combine corpora in order to maximize performance, and to understand why the specific combination of corpora works better than 1The notion of “factuality" is closely related to the notion of “belief" as used in cognitive science and AI; they differ only in the case of lying, where the author presents propositions as facts contrary to what she actually believes. See (Prabhakaran et al., 2015) for a fuller discussion. others. We choose a single resource to focus on so that we can gain insights by performing a careful study, and we choose FactBank (Saurí and Puste- jovsky, 2009) because it is one of the first carefully constructed datasets for factuality prediction. We show how insights gained from working with Fact- Bank can be used to improve performance on the CB corpus (de Marneffe et al., 2019). There are three main contributions of this work: (i) The FactBank data is complex. In order to fa- cilitate NLP research, a machine learning-friendly projection from FactBank had been previously cre- ated and widely used. We correct an error in this projection (this data set will be made available). The error means that all recent results on FactBank are …

Content truncated to 500 words