ENPAR:Enhancing Entity and Entity Pair Representations for Joint Entity Relation Extraction

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2877–2887 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2877 ENPAR:Enhancing Entity and Entity Pair Representations for Joint Entity Relation Extraction Yijun Wang1, 2, Changzhi Sun4, Yuanbin Wu3, Hao Zhou4, Lei Li4, and Junchi Yan1, 2 1Department of Computer Science and Engineering, Shanghai Jiao Tong University 2MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University 3School of Computer Science and Technology, East China Normal University 4ByteDance, AI Lab yijunwang.cs@gmail.com ybwu@cs.ecnu.edu.cn yanjunchi@sjtu.edu.cn {sunchangzhi, zhouhao.nlp, lileilab}@bytedance.com Abstract Current state-of-the-art systems for joint en- tity relation extraction (Luan et al., 2019; Wad- den et al., 2019) usually adopt the multi-task learning framework. However, annotations for these additional tasks such as coreference res- olution and event extraction are always equally hard (or even harder) to obtain. In this work, we propose a pre-training method ENPAR to improve the joint extraction performance. EN- PAR requires only the additional entity anno- tations that are much easier to collect. Unlike most existing works that only consider incor- porating entity information into the sentence encoder, we further utilize the entity pair in- formation. Specifically, we devise four novel objectives, i.e., masked entity typing, masked entity prediction, adversarial context discrim- ination, and permutation prediction, to pre- train an entity encoder and an entity pair en- coder. Comprehensive experiments show that the proposed pre-training method achieves sig- nificant improvement over BERT on ACE05, SciERC, and NYT, and outperforms current state-of-the-art on ACE05. 1 Introduction Joint extraction of entities and relations is a fun- damental task in information extraction, it aims to extract entities and relations with a unified model. Current approaches (Luan et al., 2019; Wadden et al., 2019) usually adopt the multi-task learning framework that optimizes many objectives simul- taneously, including entity recognition, relation extraction, coreference resolution, and event ex- traction. However, as large-scale manually labeled data required by these methods is unavailable in many domains, their applicability is severely re- stricted. Therefore, we expect to catch or even surpass the multi-task based joint models with less annotation cost. Compared with the annotations of coreference resolution and event extraction, entity Transformer Input MLM NSP (a) BERT Transformer Input Entity Encoder Entity Pair Encoder ACD PP MET MEP (b) ENPAR Figure 1: The network architectures and objectives of BERT and ENPAR. Our work introduce an entity en- coder and an entity pair encoder. MLM = masked lan- guage model, NSP = next sentence prediction, MET = masked entity typing, MEP = masked entity predic- tion, ACD = adversarial context discrimination, PP = permutation prediction. annotations can be easily obtained through auto- matic NER annotation tools (e.g., spaCy 1). In this paper, we focus on improving the model’s perfor- mance with just extra entity annotations. Although pre-trained models, like BERT, have shown impressive performance in many down- stream tasks, they have mainly two limitations when applied in the joint entity relation extrac- tion task. One is that currently pre-trained ob- …

Content truncated to 500 words