PairSpanBERT: An Enhanced Language Model for Bridging Resolution

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6931–6946 July 9-14, 2023 ©2023 Association for Computational Linguistics PAIRSPANBERT: An Enhanced Language Model for Bridging Resolution Hideo Kobayashi1, Yufang Hou2 and Vincent Ng1 1 Human Language Technology Research Institute, University of Texas at Dallas, USA 2 IBM Research Europe, Ireland {hideo,vince}@hlt.utdallas.edu yhou@ie.ibm.com Abstract We present PAIRSPANBERT, a SPANBERT- based pre-trained model specialized for bridg- ing resolution. PAIRSPANBERT is pre-trained with a novel objective that aims to learn the contexts in which two mentions are implic- itly linked to each other from a large amount of data automatically generated either heuristi- cally or via distance supervision with a knowl- edge graph. Despite the noise inherent in the automatically generated data, we achieve the best results reported to date on three evaluation datasets for bridging resolution when replacing SPANBERT with PAIRSPANBERT in a state- of-the-art resolver that jointly performs entity coreference resolution and bridging resolution. 1 Introduction Bridging is essential for establishing coherence among the entities within a text through non- identical semantic or encyclopedic relations (Clark, 1975; Prince, 1981). As demonstrated in Example 1, local coherence is established via the implicit link between the bridging anaphor (prices) and its antecedent (meat, milk and grain). (1) In June, farmers held onto meat, milk and grain, waiting for July’s usual state directed price rises. The Communists froze prices instead. The task of bridging resolution, which involves identifying all the bridging anaphors in a text and linking them to their antecedents, is crucial for machine comprehension of the relations between discourse entities for various downstream applica- tions, such as question answering (Anantha et al., 2021) and dialogue systems (Tseng et al., 2021). The most successful natural language learning paradigm to date is arguably the “pre-train and fine- tune" paradigm, where a model is first pre-trained on very large amounts of data in a task-agnostic, self-supervised manner and then fine-tuned using a potentially small amount of task-specific training data in the usual supervised manner. This paradigm is ideally applicable to bridging resolution, where the amount of annotated training data is relatively small, especially in comparison to the related task of entity coreference resolution. In fact, by using SPANBERT (Joshi et al., 2020) to encode the input and fine-tuning it using bridging-annotated data, Kobayashi et al. (2022b) have managed to achieve the best results reported to date on two commonly- used evaluation datasets for bridging resolution, namely ISNotes (Markert et al., 2012) and BASHI (Rösiger, 2018). A natural question is: how can we build upon the successes of this pre-train and fine-tune framework for bridging resolution? Apart from achieving state- of-the-art results, Kobayashi et al. (2022b) show that bridging resolution performance deteriorates when SPANBERT is replaced with BERT (Devlin et al., 2019) as the encoder. While it is perhaps not surprising that SPANBERT achieves better res- olution results than BERT given its superior per- formance on a wide variety of natural language processing tasks, it is important to …

Content truncated to 500 words