Coreferential Reasoning Learning for Language Representation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7170–7186, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7170 Coreferential Reasoning Learning for Language Representation Deming Ye1,2, Yankai Lin4, Jiaju Du1,2, Zhenghao Liu1,2, Peng Li4, Maosong Sun1,3, Zhiyuan Liu1,3 1Department of Computer Science and Technology, Tsinghua University, Beijing, China Institute for Artificial Intelligence, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology 2State Key Lab on Intelligent Technology and Systems, Tsinghua University, Beijing, China 3Beijing Academy of Artificial Intelligence 4Pattern Recognition Center, WeChat AI, Tencent Inc. ydm18@mails.tsinghua.edu.cn Abstract Language representation models such as BERT could effectively capture contextual se- mantic information from plain text, and have been proved to achieve promising results in lots of downstream NLP tasks with appropri- ate fine-tuning. However, most existing lan- guage representation models cannot explicitly handle coreference, which is essential to the coherent understanding of the whole discourse. To address this issue, we present CorefBERT, a novel language representation model that can capture the coreferential relations in context. The experimental results show that, compared with existing baseline models, CorefBERT can achieve significant improvements consis- tently on various downstream NLP tasks that require coreferential reasoning, while main- taining comparable performance to previous models on other common NLP tasks. The source code and experiment details of this pa- per can be obtained from https://github. com/thunlp/CorefBERT. 1 Introduction Recently, language representation models such as BERT (Devlin et al., 2019) have attracted consid- erable attention. These models usually conduct self-supervised pre-training tasks over large-scale corpus to obtain informative language representa- tion, which could capture the contextual semantic of the input text. Benefiting from this, language rep- resentation models have made significant strides in many natural language understanding tasks includ- ing natural language inference (Zhang et al., 2020), sentiment classification (Sun et al., 2019b), ques- tion answering (Talmor and Berant, 2019), relation extraction (Peters et al., 2019), fact extraction and verification (Zhou et al., 2019), and coreference resolution (Joshi et al., 2019). However, existing pre-training tasks, such as masked language modeling, usually only require models to collect local semantic and syntactic infor- mation to recover the masked tokens. Hence, lan- guage representation models may not well model the long-distance connections beyond sentence boundary in a text, such as coreference. Previous work has shown that the performance of these mod- els is not as good as human performance on the tasks requiring coreferential reasoning (Paperno et al., 2016; Dasigi et al., 2019), and they can be further improved on long-text tasks with external coreference information (Cheng and Erk, 2020; Xu et al., 2020; Zhao et al., 2020). Coreference occurs when two or more expressions in a text refer to the same entity, which is an important element for a coherent understanding of the whole discourse. For example, for comprehending the whole context of “Antoine published The Little Prince in 1943. The book follows a young prince who visits various planets in space.”, we must realize that The book refers to The Little …

Content truncated to 500 words