ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3350–3363 August 1–6, 2021. ©2021 Association for Computational Linguistics 3350 ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning Yujia Qin♣♠♦, Yankai Lin♦, Ryuichi Takanobu♣♦, Zhiyuan Liu♣∗, Peng Li♦, Heng Ji♠∗, Minlie Huang♣, Maosong Sun♣, Jie Zhou♦ ♣Department of Computer Science and Technology, Tsinghua University, Beijing, China ♠University of Illinois at Urbana-Champaign ♦Pattern Recognition Center, WeChat AI, Tencent Inc. yujiaqin16@gmail.com Abstract Pre-trained Language Models (PLMs) have shown superior performance on various down- stream Natural Language Processing (NLP) tasks. However, conventional pre-training ob- jectives do not explicitly model relational facts in text, which are crucial for textual under- standing. To address this issue, we propose a novel contrastive learning framework ERICA to obtain a deep understanding of the entities and their relations in text. Specifically, we de- fine two novel pre-training tasks to better un- derstand entities and relations: (1) the entity discrimination task to distinguish which tail entity can be inferred by the given head en- tity and relation; (2) the relation discrimination task to distinguish whether two relations are close or not semantically, which involves com- plex relational reasoning. Experimental results demonstrate that ERICA can improve typical PLMs (BERT and RoBERTa) on several lan- guage understanding tasks, including relation extraction, entity typing and question answer- ing, especially under low-resource settings.1 1 Introduction Pre-trained Language Models (PLMs) (Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019) have shown superior performance on various Natural Language Processing (NLP) tasks such as text clas- sification (Wang et al., 2018), named entity recog- nition (Sang and De Meulder, 2003), and question answering (Talmor and Berant, 2019). Benefiting from designing various effective self-supervised learning objectives, such as masked language mod- eling (Devlin et al., 2018), PLMs can effectively capture the syntax and semantics in text to gener- ate informative language representations for down- stream NLP tasks. ∗Corresponding author. 1Our code and data are publicly available at https:// github.com/thunlp/ERICA. [1] Culiacán is a city in northwestern Mexico. [2] Culiacán is the capital of the state of Sinaloa. [3] Culiacán is also the seat of Culiacán Municipality. [4] It had an urban population of 785,800 in 2015 while 905,660 lived in the entire municipality. [5] While Culiacán Municipality has a total area of 4,758 k!!, Culiacán itself is considerably smaller, measuring only. [6] Culiacán is a rail junction and is located on the Panamerican Highway that runs south to Guadalajara and Mexico City. [7] Culiacán is connected to the north with Los Mochis, and to the south with Mazatlán, Tepic. Culiacán Q: where is Guadalajara? Culiacán Mexico Panamerican Highway city of south to locate on A: Mexico. Culiacán Municipality Sinaloa Guadalajara Mexico City Los Mochis Figure 1: An example for a document “Culiacán”, in which all entities are underlined. We show entities and their relations as a relational graph, and highlight the important entities and relations to …

Content truncated to 500 words