Learning from Context or Names? An Empirical Study on Neural Relation Extraction

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3661–3672, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3661 Learning from Context or Names? An Empirical Study on Neural Relation Extraction Hao Peng1∗, Tianyu Gao2∗, Xu Han1, Yankai Lin3, Peng Li3, Zhiyuan Liu1† , Maosong Sun1, Jie Zhou3 1Department of Computer Science and Technology, Tsinghua University, Beijing, China 2Princeton University, Princeton, NJ, USA 3Pattern Recognition Center, WeChat AI, Tencent Inc, China {h-peng17,hanxu17}@mails.tsinghua.edu.cn, tianyug@princeton.edu Abstract Neural models have achieved remarkable suc- cess on relation extraction (RE) benchmarks. However, there is no clear understanding which type of information affects existing RE models to make decisions and how to further improve the performance of these models. To this end, we empirically study the effect of two main information sources in text: textual con- text and entity mentions (names). We find that (i) while context is the main source to sup- port the predictions, RE models also heavily rely on the information from entity mentions, most of which is type information, and (ii) ex- isting datasets may leak shallow heuristics via entity mentions and thus contribute to the high performance on RE benchmarks. Based on the analyses, we propose an entity-masked con- trastive pre-training framework for RE to gain a deeper understanding on both textual con- text and type information while avoiding rote memorization of entities or use of superficial cues in mentions. We carry out extensive ex- periments to support our views, and show that our framework can improve the effectiveness and robustness of neural models in different RE scenarios. All the code and datasets are released at https://github.com/thunlp/ RE-Context-or-Names. 1 Introduction Relation extraction (RE) aims at extracting rela- tional facts between entities from text, e.g., ex- tracting the fact (SpaceX, founded by, Elon Musk) from the sentence in Figure 1. Utilizing the structured knowledge captured by RE, we can construct or complete knowledge graphs (KGs), and eventually support downstream applications like question answering (Bordes et al., 2014), di- alog systems (Madotto et al., 2018) and search ∗ Equal contribution † Corresponding author e-mail: liuzy@tsinghua.edu.cn SpaceX was founded in 2002 by Elon Musk . founded by Type: organization ID: Q193701 Other info: country: US product: Falcon … Type: person ID: Q317521 Other info: citizenship: US occupation: entrepreneur … Figure 1: An example for the information provided by textual context and entity mentions in a typical RE sce- nario. From mentions, we can acquire type information and link entities to KGs, and access further knowledge about them. The IDs in the figure are from Wikidata. engines (Xiong et al., 2017). With the recent ad- vance of deep learning, neural relation extraction (NRE) models (Socher et al., 2012; Liu et al., 2013; Baldini Soares et al., 2019) have achieved the latest state-of-the-art results and some of them are even comparable with human performance on several public RE benchmarks. The success of NRE models on current RE benchmarks makes us wonder which type of in- formation these models actually grasp to help …

Content truncated to 500 words