Extracting Victim Counts from Text

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1925–1940 May 2-6, 2023 ©2023 Association for Computational Linguistics Extracting Victim Counts from Text Mian Zhong Shehzaad Dhuliawala Niklas Stoehr Institute for Machine Learning, ETH Zürich mzhong@ethz.ch shehzaad.dhuliawala@inf.ethz.ch niklas.stoehr@inf.ethz.ch Abstract Decision-makers in the humanitarian sector rely on timely and exact information during cri- sis events. Knowing how many civilians were injured during an earthquake is vital to allocate aids properly. Information about such victim counts is often only available within full-text event descriptions from newspapers and other reports. Extracting numbers from text is chal- lenging: numbers have different formats and may require numeric reasoning. This renders purely string matching-based approaches insuf- ficient. As a consequence, fine-grained counts of injured, displaced, or abused victims beyond fatalities are often not extracted and remain un- seen. We cast victim count extraction as a ques- tion answering (QA) task with a regression or classification objective. We compare regex, de- pendency parsing, semantic role labeling-based approaches, and advanced text-to-text models. Beyond model accuracy, we analyze extraction reliability and robustness which are key for this sensitive task. In particular, we discuss model calibration and investigate few-shot and out-of- distribution performance. Ultimately, we make a comprehensive recommendation on which model to select for different desiderata and data domains. Our work is among the first to apply numeracy-focused large language models in a real-world use case with a positive impact.1 1 Introduction Timely and accurate information during crisis events is crucial for rescue operations and the allo- cation of humanitarian aid (Lepuschitz and Stoehr, 2021). However, crisis information is often scarce, subjective, or biased, which renders reported num- bers in text extremely important (Hellmeier et al., 2018; Zavarella et al., 2020; Radford, 2021). For instance, the count of injured or missing people provides quantitative information about the catas- trophic impact of an earthquake. In this work, we 1Code is available online at: https://github.com/mianzg/victim_counts focus on human victims in crisis events, e.g., fatal- ities in floods, herein referred to as victim counts. A reliable estimate of victim counts is helpful dur- ing crisis (Darcy and Hofmann, 2003; Kreutzer et al., 2020), and also post-crisis, benefiting re- search to diversify measures of crisis intensity. As of now, most intensity measures are either limited to event types (Vincent, 1979; Goldstein, 1992), fa- tality counts (Kalyvas, 2006; Chaudoin et al., 2017) or both (Stoehr et al., 2022). More fine-grained measures such as injured, displaced, or abused vic- tims are not captured in most popular databases and remain unmonitored (Krause, 2013; Cruyff et al., 2017; Cullen et al., 2021). Many victim counts are reported in full-text form within event descriptions in news media. This makes their systematic collection and anal- ysis technically complex. Manual extraction of victim counts from text is very labor-intensive and does not scale to big data collections (Schrodt and Ulfelder, 2016; Lewis et al., 2016). Computer- ized approaches such as the event coding software Tabari (Schrodt, 2009) and Petrarch2 (Norris et al., 2017) …

Content truncated to 500 words