Event Detection and Factuality Assessment with Non-Expert Supervision

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1643–1648, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Event Detection and Factuality Assessment with Non-Expert Supervision Kenton Lee†, Yoav Artzi‡∗, Yejin Choi†, and Luke Zettlemoyer† † Computer Science & Engineering, University of Washington, Seattle, WA 98195 {kentonl, yejin, lsz}@cs.washington.edu ‡ Dept. of Computer Science and Cornell Tech, Cornell University, New York, NY 10011 yoav@cs.cornell.edu Abstract Events are communicated in natural lan- guage with varying degrees of certainty. For example, if you are “hoping for a raise,” it may be somewhat less likely than if you are “expecting” one. To study these distinctions, we present scalable, high- quality annotation schemes for event de- tection and fine-grained factuality assess- ment. We find that non-experts, with very little training, can reliably provide judgments about what events are men- tioned and the extent to which the author thinks they actually happened. We also show how such data enables the develop- ment of regression models for fine-grained scalar factuality predictions that outper- form strong baselines. 1 Introduction Interpretation of events—determining what the author claims did or did not happen—is impor- tant for many NLP applications, such as news arti- cle summarization or biomedical information ex- traction. However, detecting events and assessing their factuality is challenging. For example, while most non-copular verbs are events, words in gen- eral vary with use (e.g. “trade route” vs “trade with Iraq”). Events also have widely varying, context-dependent factuality cues, such as event interactions (e.g. “prevent easy access) and cue words (e.g. “ordered to” vs. “expected to”). As shown in Figure 1, these are common challenges that a model of event factuality must address. In this paper, we present new data and mod- els for these tasks, demonstrating that non-experts can provide high-quality annotations which en- able fine-grained, scalar judgments of factuality. Unlike previous work, we do not use a detailed ∗Work done at the University of Washington. (1) U.S. embassies and military installations around the world were ordered(3.0) to set(2.6) up barriers and tighten(2.6) security to prevent(1.8) easy access(-2.4) by unauthorized people. (2) Intel’s most powerful computer chip has flaws that could delay(0.8) several computer makers’ marketing efforts(2.6), but the “bugs” aren’t expected(-2.6) to hurt(-2.0) Intel. (3) President Bush on Tuesday said(3.0) the United States may extend(1.6) its naval quarantine(2.6) to Jordan’s Red Sea port of Aqaba to shut(1.4) off Iraq’s last unhindered trade route. (4) He also said(3.0) of trade(-0.8) with Iraq: “There are no shipments at the moment.” Figure 1: Example annotations with italicized event mentions and crowdsourced scalar factuality values u ∈ [−3.0, 3.0]. Positive (or negative) val- ues indicate the extent to which the author claims the events happened (or not). specification of exactly what events and factual- ity classes should be. Instead, we simply ask non-experts to find words describing things that the author claims could have happened, and rate each possibility on a scale of -3 (certainly did not happen) to 3 (certainly did). Figure 1 shows …

Content truncated to 500 words