A New Dataset and Evaluation for Belief/Factuality

Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics (*SEM 2015), pages 82–91, Denver, Colorado, June 4–5, 2015. A New Dataset and Evaluation for Belief/Factuality Vinodkumar Prabhakaran1, Tomas By2, Julia Hirschberg1, Owen Rambow3*, Samira Shaikh4, Tomek Strzalkowski4, Jennifer Tracey5, Michael Arrigo5, Rupayan Basu6, Micah Clark2, Adam Dalton2, Mona Diab7, Louise Guthrie2, Anna Prokofieva1, Stephanie Strassel5, Gregory Werner7, Janyce Wiebe8, Yorick Wilks2 1Department of Computer Science, Columbia University, New York, NY, USA 2Florida Institute for Human and Machine Cognition (IHMC), FL, USA 3Center for Computational Learning Systems, Columbia University, New York, NY, USA 4State University of New York - University of Albany, NY, USA 5Linguistic Data Consortium (LDC), University of Pennsylvania, PA, USA 6Amazon.com Inc., CA, USA 7George Washington University, DC, USA 8University of Pittsburgh, PA, USA *corresponding author; email address: rambow@ccls.columbia.edu Abstract The terms “belief” and “factuality” both re- fer to the intention of the writer to present the propositional content of an utterance as firmly believed by the writer, not firmly believed, or having some other status. This paper presents an ongoing annotation effort and an associated evaluation. 1 Introduction This paper presents an ongoing project aimed at de- veloping a community-wide evaluation of expressed belief, also known as “factuality”. Belief and fac- tuality are closely related to hedging, veridical- ity, and modality. The project has grown out of the DARPA DEFT project; participants include the Linguistic Data Consortium (LDC) and three per- former sites: Columbia University/George Wash- ington University, the Florida Institute for Human and Machine Cognition, and the University of Al- bany. The goal of our research project is not lin- guistic annotation, but the identification of meaning which is expressed in a non-linguistic manner. Such a meaning representation is useful for many applica- tions; in our project we are specifically interested in knowledge base population. A different part of the DEFT program is concerned with the representation of propositional meaning, following the tradition of the ACE program in representing entities, relations and events (ERE) (Doddington et al., 2004). The work presented here is concerned with the attitude of agents towards propositional content: do the agents express a committed belief or a non-committed be- lief in the propositional content? Our work has sev- eral characteristics that set it apart from other work: we are interested in annotation which can be done fairly quickly; we are not interested in annotating linguistic elements (such as trigger words); and we are planning an integration with sentiment annota- tion. The structure of the paper is as follows: we start out by situating our notion of “belief” with respect to other notions of extra-propositional meaning (Sec- tion 2); we then present our annotation in some de- tail, with a special comparison to FactBank (Saur´ı and Pustejovsky, 2012). While the goal of this paper is not to talk about computational systems that were run as part of the evaluation (different publications will be available for that purpose), we quickly sum- marize their main characteristics so that the evalu- ation results can …

Content truncated to 500 words