Coreference Resolution through a seq2seq Transition-Based System

Coreference Resolution through a seq2seq Transition-Based System Bernd Bohnet1, Chris Alberti2, Michael Collins2 1Google Research, The Netherlands 2Google Research, USA {bohnetbd,chrisalberti,mjcollins}@google.com Abstract Most recent coreference resolution systems use search algorithms over possible spans to identify mentions and resolve coreference. We instead present a coreference resolution sys- tem that uses a text-to-text (seq2seq) paradigm to predict mentions and links jointly. We im- plement the coreference system as a transition system and use multilingual T5 as an underly- ing language model. We obtain state-of-the-art accuracy on the CoNLL-2012 datasets with 83.3 F1-score for English (a 2.3 higher F1-score than previous work [Dobrovolskii, 2021]) using only CoNLL data for training, 68.5 F1-score for Arabic (+4.1 higher than previous work), and 74.3 F1-score for Chinese (+5.3). In addition we use the SemEval-2010 data sets for experiments in the zero-shot set- ting, a few-shot setting, and supervised setting using all available training data. We obtain substantially higher zero-shot F1-scores for 3 out of 4 languages than previous approaches and significantly exceed previous supervised state-of-the-art results for all five tested lan- guages. We provide the code and models as open source.1 1 Introduction There has been a great deal of recent research in pretrained language models that employ encoder- decoder or decoder-only architectures (e.g., see GPT-3, GLAM, Lamda [Brown et al., 2020; Du et al., 2021; Thoppilan et al., 2022]), and that can generate text using autoregressive or text-to-text (seq2seq) models (e.g., see T5, MT5 [Raffel et al., 2019; Xue et al., 2021]). These models have led to remarkable results on a number of problems. Coreference resolution is the task of finding referring expressions in text that point to the same entity in the real world. Coreference resolution is a core task in NLP, relevant to a wide range of 1https://github.com/google-research /google-research/tree/master/coref_mt5. applications (e.g., see Jurafsky and Martin [2021] Chapter 21 for discussion), but somewhat surpris- ingly, there has been relatively limited work on coreference resolution using encoder-decoder or decoder-only architectures. The state-of-the-art models on coreference problems are based on encoder-only models, such as BERT (Devlin et al., 2019) or SpanBERT (Joshi et al., 2020). All recent state-of-the-art coreference models (see Table 2), however, have the disadvan- tage of a) requiring engineering of a specialized search or structured prediction step for corefer- ence resolution, on top of the encoder’s output representations; b) often requiring a pipelined ap- proach with intermediate stages of prediction (e.g., mention detection followed by coreference predic- tion); and c) an inability to leverage more recent work in pretrained seq2seq models. This paper describes a text-to-text (seq2seq) ap- proach to coreference resolution that can directly leverage modern encoder-decoder or decoder-only models. The method takes as input a sentence at a time, together with prior context, encoded as a string, and makes predictions corresponding to coreference links. The method has the following advantages over previous approaches: • Simplicity: We use greedy seq2seq predic- tion without a separate mention detection step and do not employ a higher order decoder to identify links. • Accuracy: The …

Content truncated to 500 words