Multi-stage Pre-training over Simplified Multimodal Pre-training Models

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 220–231 August 1–6, 2021. ©2021 Association for Computational Linguistics 220 UNIRE: A Unified Label Space for Entity Relation Extraction Yijun Wang∗1, 2, Changzhi Sun∗4, Yuanbin Wu3, Hao Zhou4, Lei Li4, and Junchi Yan†1, 2 1Department of Computer Science and Engineering, Shanghai Jiao Tong University 2MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University 3School of Computer Science and Technology, East China Normal University 4ByteDance AI Lab {yjwang.cs, yanjunchi}@sjtu.edu.cn ybwu@cs.ecnu.edu.cn {sunchangzhi, zhouhao.nlp, lileilab}@bytedance.com Abstract Many joint entity relation extraction models setup two separated label spaces for the two sub-tasks (i.e., entity detection and relation classification). We argue that this setting may hinder the information interaction between en- tities and relations. In this work, we propose to eliminate the different treatment on the two sub-tasks’ label spaces. The input of our model is a table containing all word pairs from a sen- tence. Entities and relations are represented by squares and rectangles in the table. We ap- ply a unified classifier to predict each cell’s la- bel, which unifies the learning of two sub-tasks. For testing, an effective (yet fast) approximate decoder is proposed for finding squares and rectangles from tables. Experiments on three benchmarks (ACE04, ACE05, SciERC) show that, using only half the number of parameters, our model achieves competitive accuracy with the best extractor, and is faster. 1 Introduction Extracting structured information from plain texts is a long-lasting research topic in NLP. Typically, it aims to recognize specific entities and relations for profiling the semantic of sentences. An example is shown in Figure 1, where a person entity “David Perkins” and a geography entity “California” have a physical location relation PHYS. Methods for detecting entities and relations can be categorized into pipeline models or joint models. In the pipeline setting, entity models and relation models are independent with disentangled feature spaces and output label spaces. In the joint setting, on the other hand, some parameter sharing of fea- ture spaces (Miwa and Bansal, 2016; Katiyar and ∗Equal contribution. †Corresponding Author. PER PER ⊥ ⊥ PER- SOC ⊥ ⊥ ⊥ ⊥ PHYS PER PER ⊥ ⊥ PER- SOC ⊥ ⊥ ⊥ ⊥ PHYS ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ PER PER- SOC ⊥ ⊥ ⊥ ⊥ ⊥ PER- SOC PER- SOC ⊥ PER- SOC PER ⊥ ⊥ ⊥ ⊥ PHYS ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ GPE ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ORG- AFF PER ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ ⊥ GPE David Perkins and his wife are village doctors California David Perkins and his wife are California in in village doctors PER PER PER PER GPE GPE PER-SOC PER-SOC ORG-AFF PHYS PHYS Figure 1: Example of a table …

Content truncated to 500 words