Few-Shot Upsampling for Protest Size Detection

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3713–3720 August 1–6, 2021. ©2021 Association for Computational Linguistics 3713 Few-Shot Upsampling for Protest Size Detection Andrew Halterman Massachusetts Institute of Technology ahalt@mit.edu Benjamin J. Radford UNC Charlotte benjamin.radford@uncc.edu Abstract We propose a new task and dataset for a common problem in social science research: “upsampling” coarse document labels to fine- grained labels or spans. We pose the problem in a question answering format, with the an- swers providing the fine-grained labels. We provide a benchmark dataset and baselines on a socially impactful task: identifying the exact crowd size at protests and demonstra- tions in the United States given only order- of-magnitude information about protest atten- dance, a very small sample of fine-grained ex- amples, and English-language news text. We evaluate several baseline models, including zero-shot results from rule-based and question- answering models, few-shot models fine-tuned on a small set of documents, and weakly su- pervised models using a larger set of coarsely- labeled documents. We find that our rule- based model initially outperforms a zero-shot pre-trained transformer language model but that further fine-tuning on a very small subset of 25 examples substantially improves out-of- sample performance. We also demonstrate a method for fine-tuning the transformer span on only the coarse labels that performs similarly to our rule-based approach. This work will contribute to social scientists’ ability to gener- ate data to understand the causes and successes of collective action. 1 Introduction A common data collection task in social science is applying fine-grained labels to documents, includ- ing extracting specific passages from text. In many cases, social scientists already have many coarsely- labeled documents and a small number of hand- annotated documents. An automated technique for “upsampling” from coarse labels to more detailed information could help researchers produce better tailored datasets. However, this process does not fit the tools that applied researchers have access to: OWOSSO� --� On� Saturday,� supporters� of� Bernie Sanders�held�the�first�of�two rallies�at�City�Hall�in anticipation� of� Michigan's� presidential� primary election� Tuesday.� The� rally� featured� a� crowd� of roughly�30�to�40�people�and�kicked�off�at�2�p.m. In� 2016's� presidential� primary,� Sanders� beat Hillary�Clinton�by�a�slim�margin�of�49.8. Coarse Label: size category 1 (10–100 attendees) Gold Span: "30 to 40" Figure 1: Documents in our corpus have “coarse labels” reporting the order of magnitude of the protest size and “gold spans” reporting the exact size of the protest. The frequency of number words (in bold) shows why this task is not trivial. training a document classifier on coarse labels will not produce the fine-grained answers. Innovations in zero-shot and few-shot classifiers and informa- tion extraction (IE) techniques show promise, but new methods are required that can also draw on the existing coarse document annotations to improve fine-grained extraction. We introduce a new task and dataset for improv- ing information extraction systems’ performance when given many coarsely-labeled documents and a small number of documents annotated with the spans of interest.1 We draw on a dataset on dissent and collective action (hereafter, “protests”) in the United States compiled by the Crowd Counting Consortium (2020) (CCC) …

Content truncated to 500 words