Unleash GPT-2 Power for Event Detection

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6271–6282 August 1–6, 2021. ©2021 Association for Computational Linguistics 6271 Unleash GPT-2 Power for Event Detection Amir Pouran Ben Veyseh1, Viet Dac Lai1, Franck Dernoncourt2, and Thien Huu Nguyen1 1 Department of Computer and Information Science, University of Oregon, Eugene, OR 97403, USA 2 Adobe Research, San Jose, CA, USA {apouranb,vietl,thien}@cs.uoregon.edu, franck.dernoncourt@adobe.com Abstract Event Detection (ED) aims to recognize men- tions of events (i.e., event triggers) and their types in text. Recently, several ED datasets in various domains have been proposed. How- ever, the major limitation of these resources is the lack of enough training data for individual event types which hinders the efficient train- ing of data-hungry deep learning models. To overcome this issue, we propose to exploit the powerful pre-trained language model GPT-2 to generate training samples for ED. To pre- vent the noises inevitable in automatically gen- erated data from hampering training process, we propose to exploit a teacher-student archi- tecture in which the teacher is supposed to learn anchor knowledge from the original data. The student is then trained on combination of the original and GPT-generated data while being led by the anchor knowledge from the teacher. Optimal transport is introduced to fa- cilitate the anchor knowledge-based guidance between the two networks. We evaluate the proposed model on multiple ED benchmark datasets, gaining consistent improvement and establishing state-of-the-art results for ED. 1 Introduction An important task of Information Extraction (IE) involves Event Detection (ED) whose goal is to recognize and classify words/phrases that evoke events in text (i.e., event triggers). For instance, in the sentence “The organization donated 2 mil- lion dollars to humanitarian helps.”, ED systems should recognize “donated” as an event trigger of type Pay. We differentiate two subtasks in ED, i.e., Event Identification (EI): a binary classifica- tion problem to predict if a word in text is an event trigger or not, and Event Classification (EC): a multi-class classification problem to classify event triggers according to predefined event types. Several methods have been introduced for ED, extending from feature-based models (Ahn, 2006; Liao and Grishman, 2010a; Miwa et al., 2014) to advanced deep learning methods (Nguyen and Gr- ishman, 2015; Chen et al., 2015; Nguyen et al., 2016c; Sha et al., 2018; Zhang et al., 2020b; Nguyen et al., 2021). Although deep learning mod- els have achieved substantial improvement, their requirement of large training datasets together with the small sizes of existing ED datasets constitutes a major hurdle to build high-performing ED models. Recently, there have been some efforts to enlarge training data for ED models by exploiting unsu- pervised (Huang et al., 2016; Yuan et al., 2018) or distantly-supervised (Keith et al., 2017; Nguyen and Nguyen, 2018; Araki and Mitamura, 2018) techniques. The common strategy in these meth- ods is to exploit unlabeled text data that are rich in event mentions to aid the expansion of training data for …

Content truncated to 500 words