Lifting the Curse of Capacity Gap in Distilling Language Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 393–409 July 9-14, 2023 ©2023 Association for Computational Linguistics What Is Overlap Knowledge in Event Argument Extraction? APE: A Cross-datasets Transfer Learning Model for EAE Kaihang Zhang1, Kai Shuang 1∗, Xinyue Yang1, Xuyang Yao2 and Jinyu Guo13 1State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications 2China Telecom Research Institute 3University of Cambridge {zkh1999, shuangk, crescent3919, guojinyu}@bupt.edu.cn yaoxy11@chinatelecom.cn Abstract The EAE task extracts a structured event record from an event text. Most existing approaches train the EAE model on each dataset indepen- dently and ignore the overlap knowledge across datasets. However, insufficient event records in a single dataset often prevent the existing model from achieving better performance. In this paper, we clearly define the overlap knowl- edge across datasets and split the knowledge of the EAE task into overlap knowledge across datasets and specific knowledge of the target dataset. We propose APE model to learn the two parts of knowledge in two serial learn- ing phases without causing catastrophic forget- ting. In addition, we formulate both learning phases as conditional generation tasks and de- sign Stressing Entity Type Prompt to close the gap between the two phases. The experiments show APE achieves new state-of-the-art with a large margin in the EAE task. When only ten records are available in the target dataset, our model dramatically outperforms the baseline model with average 27.27% F1 gain.1 1 Introduction Event extraction (EE) is a pivotal task in informa- tion extraction. Typically, the event extraction task can be divided into two sub-tasks: event detec- tion (ED) and event argument extraction (EAE). Thanks to recent works (Liu et al., 2022a; Sheng et al., 2022; Lai et al., 2020), event detection has achieved significant progress. The main challenge of EE lies in the EAE task. The EAE task aims to extract a structured event record from an event text. Since different datasets often have various event types and argument struc- tures, most studies (Ma et al., 2022; Lu et al., 2021; Liu et al., 2022b) train the EAE model on each dataset independently, such as ACE 2005 (Dod- dington et al., 2004), RAMS (Ebner et al., 2020), ∗Corresponding author. 1https://github.com/ZKH-1999/APE Figure 1: The illustration of overlap knowledge and WikiEvents (Li et al., 2021). However, one sin- gle dataset often cannot provide sufficient event records, which seriously prevents those models from achieving better performance. Especially in some industrial applications, the in-domain event record collection incurs expensive and time- consuming manual annotation. We argue that there is abundant transferable all-purpose knowledge of the EAE task among different datasets, called over- lap knowledge. Exploring the overlap knowledge from existing datasets can significantly improve the model’s performance and reduce the need for newly annotated data. How to transfer knowledge across datasets has yet to be well studied. Only Zhou et al. (2022) attempted to introduce variational information bot- tleneck to retain the shared knowledge between two datasets and …

Content truncated to 500 words