Examining the Impact of Coreference Resolution on Quote Attribution

Examining the Impact of Coreference Resolution on Quote Attribution Tim O’Keefe Kellie Webster James R. Curran tokeefe@ kweb3773@ james@ it.usyd.edu.au uni.sydney.edu.au it.usyd.edu.au e -lab, School of Information Technologies University of Sydney NSW 2006, Australia Abstract Quote attribution is the task of identify- ing the speaker of each quote within a document. While recent research has es- tablished large-scale corpora for this task, these corpora are not yet consistent in the way they handle candidate speakers, and many of the reported results rely on gold standard annotations of both entities and coreference chains. In this work we evaluate three quote at- tribution systems with automatically pro- duced candidate speakers and coreference chains. We perform these experiments over four separate corpora, which allows us to determine how coreference resolu- tion effects quote attribution, and to use the task as an extrinsic evaluation of three coreference systems. 1 Introduction News articles are often driven by the quotes that appear within them. Approximately 32% of the tokens in the Sydney Morning Herald Corpus (SMHC) (Pareti et al., 2013) appear within a quote. Ignoring the attributed nature of this text can result in incorrectly assigning text to a document’s au- thor, rather than to the speaker the author attributes it to. Quote attribution is thus important for appli- cations such as information retrieval, opinion min- ing, media monitoring, and others. Early research into quote attribution and quote extraction was largely rule-based, as there was no large-scale data available. Several more re- cent studies (Elson and McKeown, 2010; O’Keefe et al., 2012; He et al., 2013; Pareti et al., 2013) have addressed this with corpora covering both news articles and literature. However, despite the importance of candidate speakers to this task, work thus far has treated candidates speakers in- consistently. Elson and McKeown (2010) include automatically identified named entities and com- mon nouns, but do not include pronominal ref- erences or attempt coreference, which they state is problematic due to the domain (literature). He et al. (2013) include automatically identified named entities with limited gold-standard coref- erence, but do not include pronouns or common nouns. The SMHC (Pareti et al., 2013) includes gold-standard named entities and pronouns, as well as gold-standard coreference, but does not in- clude common noun candidates. Finally the PARC (Pareti, 2012) is intended to cover attribution more generally, and so does not include any candidate speakers except for those that have attributed text. Our work addresses the problem of inconsis- tent candidates within these corpora by separately aligning the output of three coreference resolution systems, Stanford (Lee et al., 2011), Reconcile (Stoyanov et al., 2010), and a naive baseline sys- tem, with the gold-standard speaker annotations. We can then evaluate the quote attribution meth- ods from O’Keefe et al. (2012) with a set of speak- ers that have been identified in a more consistent manner across attribution methods and corpora. O’Keefe et al. note that one of the primary fac- tors confounding their evaluation was that the set of candidates was not …

Content truncated to 500 words