Moving on from OntoNotes: Coreference Resolution Model Transfer

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5241–5256 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5241 Moving on from OntoNotes: Coreference Resolution Model Transfer Patrick Xia Johns Hopkins University paxia@cs.jhu.edu Benjamin Van Durme Johns Hopkins University vandurme@cs.jhu.edu Abstract Academic neural models for coreference reso- lution (coref) are typically trained on a single dataset, OntoNotes, and model improvements are benchmarked on that same dataset. How- ever, real-world applications of coref depend on the annotation guidelines and the domain of the target dataset, which often differ from those of OntoNotes. We aim to quantify trans- ferability of coref models based on the num- ber of annotated documents available in the tar- get dataset. We examine eleven target datasets and find that continued training is consistently effective and especially beneficial when there are few target documents. We establish new benchmarks across several datasets, including state-of-the-art results on PreCo. 1 Introduction Starting initially with neurally-learned features (Clark and Manning, 2016a,b), end-to-end neu- ral models for coreference resolution (coref) (Lee et al., 2017, 2018) have been developed and im- bued with the benefits from contextualized lan- guage modeling (Joshi et al., 2019, 2020) and ad- ditional pretraining (Wu et al., 2020). At the same time, the number of parameters used in these mod- els have increased, raising questions of overfitting our research to a specific dataset. Several studies show that fully-trained neural models on preex- isting large datasets do not transfer well to new domains (Akta¸s et al., 2020; Bamman et al., 2020; Timmapathini et al., 2021), and that rule-based baselines can still be superior (Poot and van Cra- nenburgh, 2020). Further, while prior work has analyzed fully-trained models for mention pairs, like gender bias (Rudinger et al., 2018; Webster et al., 2018; Zhao et al., 2019), there has not been a comprehensive comparison analyzing transfer across datasets for document-level coref. We bridge the current gap in understanding be- tween the strength of pretrained models in contrast to the value of annotated target data, in light of the strong few-shot capabilities demonstrated by pretrained language models (Brown et al., 2020; Schick and Schütze, 2021). While transfer in other NLP tasks have been studied more in-depth, trans- fer in coref has scarcely been examined despite recent models containing hundreds of millions of parameters. We investigate model transfer across datasets with continued training, in which a fully- trained model on a source dataset is further trained on a small number of target dataset examples (Sen- nrich et al., 2016; Khayrallah et al., 2018).1 We contribute the first study of neural coref trans- fer, showing that continued training is effective on eleven datasets spanning different domains, an- notation guidelines, and languages. We find evi- dence that OntoNotes, a widely-used but license- requiring dataset for benchmarking coref, is no better at model transfer than the freely-available PreCo. We establish modern benchmarks on sev- eral understudied datasets, including state-of-the- art results on PreCo. Additionally, we analyze practical considerations regarding model selection, catastrophic forgetting, …

Content truncated to 500 words