Corpus-Level Evaluation for Event QA: The IndiaPoliceEvents Corpus Covering the 2002 Gujarat Violence

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4240–4253 August 1–6, 2021. ©2021 Association for Computational Linguistics 4240 Corpus-Level Evaluation for Event QA: The IndiaPoliceEvents Corpus Covering the 2002 Gujarat Violence Andrew Halterman∗ Massachusetts Institute of Technology ahalt@mit.edu Katherine A. Keith∗ University of Massachusetts Amherst kkeith@cs.umass.edu Sheikh Muhammad Sarwar∗ University of Massachusetts Amherst smsarwar@cs.umass.edu Brendan O’Connor University of Massachusetts Amherst brenocon@cs.umass.edu Abstract Automated event extraction in social science applications often requires corpus-level evalu- ations: for example, aggregating text predic- tions across metadata and unbiased estimates of recall. We combine corpus-level evalu- ation requirements with a real-world, social science setting and introduce the INDIAPO- LICEEVENTS corpus—all 21,391 sentences from 1,257 English-language Times of India articles about events in the state of Gujarat during March 2002. Our trained annotators read and label every document for mentions of police activity events, allowing for unbi- ased recall evaluations. In contrast to other datasets with structured event representations, we gather annotations by posing natural ques- tions, and evaluate off-the-shelf models for three different tasks: sentence classification, document ranking, and temporal aggregation of target events. We present baseline results from zero-shot BERT-based models fine-tuned on natural language inference and passage re- trieval tasks. Our novel corpus-level eval- uations and annotation approach can guide creation of similar social-science-oriented re- sources in the future. 1 Introduction Understanding the actions taken by political actors is at the heart of political science research: How do actors respond to contested elections (Daxecker et al., 2019)? How many people attend protests (Chenoweth and Lewis, 2013)? Which religious groups are engaged in violence (Brathwaite and Park, 2018)? Why do some governments try to pre- vent anti-minority riots while others do not (Wilkin- son, 2006)? In the absence of official records, so- cial scientists often turn to news data to extract the actions of actors and surrounding events. These ∗ Indicates joint first-authorship. Figure 1: Motivation (A-B) and procedures (B-C) for this paper: A. Social scientists often use text data to an- swer substantive questions about temporal aggregates. B. To answer these questions, domain experts use natu- ral language to define semantic event classes of interest. C. Our INDIAPOLICEEVENTS dataset: Humans anno- tate every sentence in the corpus in order to evaluate whether a system achieves full recall of relevant events. In production, computational models run B’s queries to classify or rank sentences or documents, which are aggregated to answer A. news-based event datasets are often constructed by hand, requiring large investments of time and money and limiting the number of researchers who can undertake data collection efforts. Automated extraction of political events and ac- tors is already prominent in social science (Schrodt et al., 1994; King and Lowe, 2003; Hanna, 2014; Hammond and Weidmann, 2014; Boschee et al., 2015; Beieler et al., 2016; Osorio and Reyes, 2017) and is increasingly promising given recent gains in information extraction (IE), the automatic conver- sion of unstructured text to structured datasets (Gr- ishman, 1997; McCallum, 2005; Grishman, 2019). While social scientists and IE researchers …

Content truncated to 500 words