Won’t Get Fooled Again: Answering Questions with False Premises

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 508–523 July 9-14, 2023 ©2023 Association for Computational Linguistics Learning with Partial Annotations for Event Detection Jian Liu1, Dianbo Sui2, Kang Liu3, Haoyan Liu4 and Zhe Zhao4 1 Beijing Jiaotong University 2 Harbin Institute of Technology 3 The Laboratory of Cognition and Decision Intelligence for Complex Systems Institute of Automation, Chinese Academy of Sciences 4 Tencent AI Lab jianliu@bjtu.edu.cn; dianbosui@hit.edu.cn; kliu@nlpr.ia.ac.cn {haoyanliu, nlpzhezhao}@tencent.com Abstract Event detection (ED) seeks to discover and classify event instances in plain texts. Pre- vious methods for ED typically adopt super- vised learning, requiring fully labeled and high-quality training data. However, in a real- world application, we may not obtain clean training data but only partially labeled one, which could substantially impede the learning process. In this work, we conduct a semi- nal study for learning with partial annotations for ED. We propose a new trigger localization formulation using contrastive learning to dis- tinguish ground-truth triggers from contexts, showing a decent robustness for addressing partial annotation noise. Impressively, in an extreme scenario where more than 90% of events are unlabeled, our approach achieves an F1 score of over 60%. In addition, we re- annotate and make available two fully anno- tated subsets of ACE 2005 to serve as an unbi- ased benchmark for event detection. We hope our approach and data will inspire future stud- ies on this vital yet understudied problem. 1 Introduction Deep learning models have shown impressive performance in event detection (ED) since large amounts of event data have become available (Chen et al., 2015; Nguyen and Grishman, 2015). How- ever, such models require fully labelled and high- quality data — in practice, we cannot ensure that every event is identified, and as a result, we often face the partial annotation issue, as depicted in Figure 1. We show a high rate of partial annotation in real-world event datasets. For example, in ACE 2005, which is widely used as a benchmark for ED evaluation (Christopher Walker and Maeda, 2006), nearly 20% of events are not labelled (see Table 2). Using a partially labelled dataset as a fully la- belled one for training runs the risk of mis-training on false negatives, and using a partially labelled dataset for evaluation biases comparison. How- A man died when a heavy tank devastated the hotel. Gold: Partial: False Negative O O Die O O O O Attack O O O O Die O O O O O O O S1: Figure 1: The partial annotation issue in ED. The Gold row indicates ground-truth labels; the Partial row in- dicates the partial annotation case we address in this study, where the devastated event is not labeled. ever, this issue is still understudied in the existing literature (Liu, 2018; Liu et al., 2020b). In this work, we present a seminal study of learn- ing with partial annotations for ED, with contri- butions in methodology, data, and practical ap- plications. In our method, …

Content truncated to 500 words