Pre-training Entity Relation Encoder with Intra-span and Inter-span Information

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1692–1705, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1692 Pre-training Entity Relation Encoder with Intra-span and Inter-span Information Yijun Wang1, 2, Changzhi Sun3, Yuanbin Wu3, Junchi Yan1, 2, Peng Gao4, and Guotong Xie4 1Department of Computer Science and Engineering, Shanghai Jiao Tong University 2MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University 3School of Computer Science and Technology, East China Normal University 4Ping An Technology (Shenzhen) Co. Ltd. Shenzhen, China {yijunwang.cs, czsun.cs}@gmail.com ybwu@cs.ecnu.edu.cn yanjunchi@sjtu.edu.cn {gaopeng712, xieguotongg}@pingan.com.cn Abstract In this paper, we integrate span-related infor- mation into pre-trained encoder for entity rela- tion extraction task. Instead of using general- purpose sentence encoder (e.g., existing uni- versal pre-trained models), we introduce a span encoder and a span pair encoder to the pre-training network, which makes it easier to import intra-span and inter-span informa- tion into the pre-trained model. To learn the encoders, we devise three customized pre- training objectives from different perspectives, which target on tokens, spans, and span pairs. In particular, a span encoder is trained to re- cover a random shuffling of tokens in a span, and a span pair encoder is trained to predict positive pairs that are from the same sentences and negative pairs that are from different sen- tences using contrastive loss. Experimental results show that the proposed pre-training method outperforms distantly supervised pre- training, and achieves promising performance on two entity relation extraction benchmark datasets (ACE05, SciERC). 1 Introduction Extraction of entities and relations from free texts is an important task in NLP. Its goal is to recog- nize text spans with specific types (entities) and semantic relations among those entities (relations). Current state-of-the-art systems usually employ the supervised joint learning algorithm (Miwa and Bansal, 2016; Sun et al., 2018, 2019a), which can alleviate error propagation caused by the pipeline method. In this paper, we focus on joint entity relation extraction. Recently, pre-trained models (Devlin et al., 2018; Dong et al., 2019) have substantially advanced a va- riety of NLP tasks, including entity relation extrac- tion (Li et al., 2019; Wadden et al., 2019). (Wadden et al., 2019) adopt BERT as a sentence encoder and build a multi-task framework for information BERT SpanBERT ERNIE Ours Token Level ✓ ✓ ✓ ✓ Span Level ✓ ✓ ✓ Span Pair Level ✓ Sentence Level ✓ ✓ Table 1: Comparison between pre-training objectives. ✓ means that additional annotations (entities) are used. extraction. However, universal pre-trained mod- els are usually trained without explicitly handling text spans and relation among text span pairs. For example, the objectives of BERT are masked lan- guage model and next sentence prediction, which are defined at the token level and sentence level, respectively. It rarely considers incorporating span- related knowledge, which can provide rich infor- mation for better extracting entities and relations (Table 1). The traditional way to introduce more entity re- lation related information is through distant super- vision, which aligns triples in knowledge bases and free …

Content truncated to 500 words