Pre-training Mention Representations in Coreference Models

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8534–8540, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8534 Pre-training Mention Representations in Coreference Models Yuval Varkel Tel Aviv University Amir Globerson Tel Aviv University and Google Research Abstract Collecting labeled data for coreference resolu- tion is a challenging task, requiring skilled an- notators. It is thus desirable to develop coref- erence resolution models that can make use of unlabeled data. Here we provide such an ap- proach for the powerful class of neural corefer- ence models. These models rely on represen- tations of mentions, and we show these repre- sentations can be learned in a self-supervised manner towards improving resolution accu- racy. We propose two self-supervised tasks that are closely related to coreference resolu- tion and thus improve mention representation. Applying this approach to the GAP dataset re- sults in new state of the arts results. 1 Introduction Coreference resolution models cluster mentions by their referring entities. Almost all such models rely on vector representations of mentions (Clark and Manning, 2016; Lee et al., 2017, 2018; Denis and Baldridge, 2008; Rahman and Ng, 2009; Durrett et al., 2013; Chang et al., 2013; Wiseman et al., 2016; Martschat and Strube, 2015). The represen- tations for all mentions are then compared (usually sequentially) and mention pairs judged to be most similar are considered coreferent. Thus, the mention representation is a key com- ponent in modern coreference resolution models. Indeed, it has recently been shown that improving this representation leads to improved resolution per- formance. For example, BERT embeddings were used in (Joshi et al., 2019b) and SpanBERT (Joshi et al., 2019a) further improved performance. However, both BERT and SpanBERT represen- tations are trained on self-supervised tasks that seem quite distant from coreference resolution (e.g., masked-word-prediction in BERT, and masked- whole-span-prediction in SpanBERT). This sug- gests the possibility that unlabeled data can be used for further improving coreference resolution if we use self-supervised tasks that are more closely re- lated to coreference resolution. Motivated by the above, we ask: which self- supervised tasks should be used to improve men- tion representation for coreference resolution. Two recent attempts for pre-training corefer- ence models have focused on tasks such as lan- guage modeling (Liu et al., 2019) and masked- word-prediction for name resolution (Kocijan et al., 2019). Here we propose self-supervision tasks that train the coreference model directly (rather than just the underlying BERT), resulting in improved mention representations and resolution accuracy. We identify two signals in a text that are highly informative for coreference resolution and show how to use them for self-supervision. The first sig- nal is that the same name can appear multiple times in a text, and these mentions very likely corefer. Thus we can train mention representations to be similar for these mentions. The second signal is pronouns. Since each pronoun is likely to refer to some mention, we optimize mention representa- tions to maximize the accuracy of this prediction. We describe a training procedure …

Content truncated to 500 words