AmbiCoref: Evaluating Human and Model Sensitivity to Ambiguous Coreference

Findings of the Association for Computational Linguistics: EACL 2023, pages 1023–1030 May 2-6, 2023 ©2023 Association for Computational Linguistics AMBICOREF: Evaluating Human and Model Sensitivity to Ambiguous Coreference Yuewei Yuan and Chaitanya Malaviya and Mark Yatskar University of Pennsylvania {yuewei, cmalaviy, myatskar}@seas.upenn.edu Abstract Given a sentence “Abby told Brittney that she upset Courtney”, one would struggle to understand who “she” refers to, and ask for clarification. However, if the word “upset” were replaced with “hugged”, “she” unambigu- ously refers to Abby. We study if modern co-reference resolution models are sensitive to such pronominal ambiguity. To this end, we construct AMBICOREF, a diagnostic corpus of minimal sentence pairs with ambiguous and un- ambiguous referents. Our examples generalize psycholinguistic studies of human perception of ambiguity around particular arrangements of verbs and their arguments. Analysis shows that (1) humans are less sure of referents in ambigu- ous AmbiCoref examples than unambiguous ones, and (2) most coreference models show lit- tle difference in output between ambiguous and unambiguous pairs. We release AMBICOREF as a diagnostic corpus for testing whether mod- els treat ambiguity similarly to humans.1 1 Introduction Ambiguity is a fundamental feature of lan- guage (Wasow et al., 2003) that some linguists believe arises because of a pressure for efficient communication (Haywood et al., 2005; Piantadosi et al., 2012). Recently, several works have high- lighted the existence of ambiguity in tasks such as question answering (Min et al., 2020; Guo et al., 2021), frame disambiguation (Dumitrache et al., 2019), anaphora resolution (Poesio and Artstein, 2005) and language modeling (Aina and Linzen, 2021). Yet systematic evaluation of how models react to ambiguity across many types of language processing problems is missing. We contribute one such study about coreference resolution. Coreference resolution is crucial to natural lan- guage understanding, especially in long contexts, such as dialog. Ambiguity may arise naturally 1Our dataset and code is available at https://github.com/LucyYYW/AmbiCoref. in dialog, but existing models do not have well- defined target behavior for such coreferences. In contrast, when people encounter coreferential ambi- guity, they recognize it, and can ask for clarification. Existing resources, such as OntoNotes (Weischedel et al., 2013), do not provide fine-grained annota- tions of such instances to evaluate model behavior. This may result in models not being calibrated to handle the uncertainty in interpretations of ambigu- ous statements. In this work, we ask how sensitive to ambiguity are models trained on these resources? To understand how existing coreference mod- els react to ambiguity, we construct a diagnostic corpus, AMBICOREF. AMBICOREF is composed of minimal pairs with ambiguous and unambigu- ous referents, created from four types of templates. Ambiguity is achieved by reducing context sizes to one sentence, and creating sentences where par- ticipating verbs under-constrain the interpretation of their arguments. For example, in Table 1, line 2, our first template leverages ambiguity around verbs expressing subjective experiences.2 The tem- plates are designed by drawing on psycholinguistic studies (Springston, 1976; Caramazza et al., 1977; Rohde and Kehler, 2014) and a core contribution of our work …

Content truncated to 500 words