BERT for Coreference Resolution: Baselines and Analysis

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5803–5808, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5803 BERT for Coreference Resolution: Baselines and Analysis Mandar Joshi † Omer Levy§ Daniel S. Weld†ϵ Luke Zettlemoyer†§ † Allen School of Computer Science & Engineering, University of Washington, Seattle, WA {mandar90,weld,lsz}@cs.washington.edu ϵAllen Institute of Artificial Intelligence, Seattle {danw}@allenai.org § Facebook AI Research, Seattle {omerlevy,lsz}@fb.com Abstract We apply BERT to coreference resolu- tion, achieving strong improvements on the OntoNotes (+3.9 F1) and GAP (+11.5 F1) benchmarks. A qualitative analysis of model predictions indicates that, compared to ELMo and BERT-base, BERT-large is particularly better at distinguishing between related but distinct entities (e.g., President and CEO). However, there is still room for improvement in modeling document-level context, conver- sations, and mention paraphrasing. Our code and models are publicly available1. 1 Introduction Recent BERT-based models have reported dra- matic gains on multiple semantic benchmarks in- cluding question-answering, natural language in- ference, and named entity recognition (Devlin et al., 2019). Apart from better bidirectional rea- soning, one of BERT’s major improvements over previous methods (Peters et al., 2018; McCann et al., 2017) is passage-level training,2 which al- lows it to better model longer sequences. We fine-tune BERT to coreference resolution, achieving strong improvements on the GAP (Web- ster et al., 2018) and OntoNotes (Pradhan et al., 2012) benchmarks. We present two ways of ex- tending the c2f-coref model in Lee et al. (2018). The independent variant uses non-overlapping segments each of which acts as an independent instance for BERT. The overlap variant splits the document into overlapping segments so as to pro- vide the model with context beyond 512 tokens. BERT-large improves over ELMo-based c2f-coref 3.9% on OntoNotes and 11.5% on GAP (both ab- solute). 1https://github.com/mandarjoshi90/ coref 2Each BERT training example consists of around 512 word pieces, while ELMo is trained on single sentences. A qualitative analysis of BERT and ELMo- based models (Table 3) suggests that BERT-large (unlike BERT-base) is remarkably better at dis- tinguishing between related yet distinct entities or concepts (e.g., Repulse Bay and Victoria Harbor). However, both models often struggle to resolve coreferences for cases that require world knowl- edge (e.g., the developing story and the scandal). Likewise, modeling pronouns remains difficult, especially in conversations. We also find that BERT-large benefits from us- ing longer context windows (384 word pieces) while BERT-base performs better with shorter contexts (128 word pieces). Yet, both variants perform much worse with longer context windows (512 tokens) in spite of being trained on 512-size contexts. Moreover, the overlap variant, which ar- tificially extends the context window beyond 512 tokens provides no improvement. This indicates that using larger context windows for pretraining might not translate into effective long-range fea- tures for a downstream task. Larger models also exacerbate the memory-intensive nature of span representations (Lee et al., 2017), which have driven recent improvements in coreference reso- lution. Together, these observations suggest …

Content truncated to 500 words