Type B Reflexivization as an Unambiguous Testbed for Multilingual Multi-Task Gender Bias

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2637–2648, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2637 Type B Reflexivization as an Unambiguous Testbed for Multilingual Multi-Task Gender Bias Ana Valeria Gonz´alez1, Maria Barrett1,2, Rasmus Hvingelby2, Kellie Webster3, and Anders Søgaard1,3 1University of Copenhagen, 2Alexandra Instituttet, 3Google Research {ana,mjb,soegaard}@di.ku.dk, rasmus.hvingelby@alexandra.dk websterk@google.com Abstract The one-sided focus on English in previous studies of gender bias in NLP misses out on opportunities in other languages: English challenge datasets such as GAP and Wino- Gender highlight model preferences that are ”hallucinatory”, e.g., disambiguating gender- ambiguous occurrences of ’doctor’ as male doctors. We show that for languages with type B reflexivization, e.g., Swedish and Russian, we can construct multi-task challenge datasets for detecting gender bias that lead to unam- biguously wrong model predictions: In these languages, the direct translation of ’the doc- tor removed his mask’ is not ambiguous be- tween a coreferential reading and a disjoint reading. Instead, the coreferential reading re- quires a non-gendered pronoun, and the gen- dered, possessive pronouns are anti-reflexive. We present a multilingual, multi-task chal- lenge dataset, which spans four languages and four NLP tasks and focuses only on this phe- nomenon. We find evidence for gender bias across all task-language combinations and cor- relate model bias with national labor market statistics. 1 Introduction A reflexive pronoun is an anaphor that requires a c-commanding antecedent within its binding do- main (Chomsky, 1991).1 In languages with Type B reflexivization (Heine, 2005), the referent of a re- flexive possessive pronoun has to be the subject of the clause, while non-reflexive possessive pronouns (so-called anti-reflexives) trigger an interpretation where its referent is not the subject; see Table 1. We focus on the subset of those languages in which anti-reflexive possessive pronouns are gen- 1This means that the antecedent should be in the same sentence, be different from the pronoun and not command it, but any ancestor of the antecedent is an ancestor of the pronoun. This is why in Lea1’s sister2 taught herself1∗/2/3∗ the pronoun refers to sister, not to Lea or a discourse referent. TYPE A TYPE B Person 1st 2nd 3rd 1st 2nd 3rd REFL ✓ ✓ ✓ ✓ ✓ Table 1: In Type B reflexivization (Heine, 2005), 3rd person pronouns cannot be used reflexively. We are in- terested in Type B languages with gendered pronouns, and where the non-gendered special (3rd person) reflex- ive marker has a possessive form. dered, but reflexives are not. This includes Chinese, Russian, Danish, and Swedish, as well as other Scandinavian, Slavic, and Sino-Tibetan languages languages (B´ıl´y, 1981; Battistella, 1989; Kiparsky, 2001).2 Our motivation for highlighting this partic- ular linguistic phenomenon is that the antecedents of reflexive and anti-reflexive pronouns are gram- matically determined; if gender bias leads mod- els (or humans) to predict alternative coreference chains, this violates hard grammatical rules and is thus a clear case of gender bias leading not only to ’hallucinations’,3 but to errors. To see this, consider the following examples: (1) The …

Content truncated to 500 words