Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns

Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns Kellie Webster and Marta Recasens and Vera Axelrod and Jason Baldridge Google AI Language {websterk, recasens, vaxelrod, jasonbaldridge}@google.com Abstract Coreference resolution is an important task for natural language understanding, and the resolution of ambiguous pronouns a longstanding challenge. Nonetheless, ex- isting corpora do not capture ambiguous pronouns in sufficient volume or diversity to accurately indicate the practical utility of models. Furthermore, we find gender bias in existing corpora and systems fa- voring masculine entities. To address this, we present and release GAP, a gender- balanced labeled corpus of 8,908 ambigu- ous pronoun–name pairs sampled to provide diverse coverage of challenges posed by real-world text. We explore a range of baselines that demonstrate the complexity of the challenge, the best achieving just 66.9% F1. We show that syntactic struc- ture and continuous neural models provide promising, complementary cues for ap- proaching the challenge. 1 Introduction Coreference resolution involves linking referring expressions that evoke the same discourse en- tity, as defined in shared tasks such as CoNLL 2011/2012 (Pradhan et al., 2012) and MUC (Grishman and Sundheim, 1996). Unfortunately, high scores on these tasks do not necessarily translate into acceptable performance for down- stream applications such as machine translation (Guillou, 2012) and fact extraction (Nakayama, 2008). In particular, high-scoring systems suc- cessfully identify coreference relationships be- tween string-matching proper names, but fare worse on anaphoric mentions such as pronouns and common noun phrases (Stoyanov et al., 2009; Rahman and Ng, 2012; Durrett and Klein, 2013). We consider the problem of resolving gendered ambiguous pronouns in English, such as she1 in: (1) In May, Fujisawa joined Mari Motohashi’s rink as the team’s skip, moving back from Karuizawa to Kitami where she had spent her junior days. With this scope, we make three key contributions: • We design an extensible, language-independent mechanism for extracting challenging am- biguous pronouns from text. • We build and release GAP, a human-labeled corpus of 8,908 ambiguous pronoun–name pairs derived from Wikipedia.2 This data set targets the challenges of resolving naturally occurring ambiguous pronouns and rewards systems that are gender-fair. • We run four state-of-the-art coreference re- solvers and several competitive simple base- lines on GAP to understand limitations in current modeling, including gender bias. We find that syntactic structure and trans- former models (Vaswani et al., 2017) pro- vide promising, complementary cues for approaching GAP. Coreference resolution decisions can drastically alter how automatic systems process text. Bi- ases in automatic systems have caused a wide range of underrepresented groups to be served in an inequitable way by downstream applications (Hardt, 2014). We take the construction of the new GAP corpus as an opportunity to reduce gen- der bias in coreference data sets; in this way, GAP can promote equitable modeling of reference phenomena complementary to the recent work of Zhao et al. (2018) and Rudinger et al. (2018). 1The examples throughout the paper highlight the am- biguous pronoun in bold, the two potential coreferent names in italics, and …

Content truncated to 500 words