Investigating Failures of Automatic Translationin the Case of Unambiguous Gender

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3454 - 3469 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Investigating Failures of Automatic Translation in the Case of Unambiguous Gender Adithya Renduchintala Facebook AI adirendu@fb.com Adina Williams Facebook AI Research adinawilliams@fb.com Abstract Transformer-based models are the modern work horses for neural machine translation (NMT), reaching state of the art across several benchmarks. Despite their impressive accu- racy, we observe a systemic and rudimentary class of errors made by current state-of-the-art NMT models with regards to translating from a language that doesn’t mark gender on nouns into others that do. We find that even when the surrounding context provides unambiguous ev- idence of the appropriate grammatical gender marking, no tested model was able to accu- rately gender occupation nouns systematically. We release an evaluation scheme and dataset for measuring the ability of NMT models to translate gender morphology correctly in unambiguous contexts across syntactically diverse sentences. Our dataset translates from an English source into 20 languages from several different language families. With the availability of this dataset, our hope is that the NMT community can iterate on solutions for this class of especially egregious errors. 1 Introduction Neural machine translation models are trained on vast amounts of data and consistently attain strong performance on standard benchmarks (Barrault et al., 2020). Despite this impressive achievement, state-of-the-art MT models are often largely unable to make basic deductions regarding how to correctly inflect nouns with grammatical gender. Previous work measured gender bias by determining how often models translated pronouns coreferent with stereotypical occupation noun stereotypically (e.g., Stanovsky et al. 2019; Prates et al. 2019). Crucially, in this ambiguous setting, the “correct” gender was genuinely under-determined given the context, which allowed for investigating the underlying (often stereotypical) “assumptions” of machine translation models (i.e., that most if not all nurses are women). However, gender mistakes in translation go beyond stereotyping: in some cases, assigning the wrong gender to a noun can result in a genuine mistranslation (i.e., a factual error). In this work, we cast the task of measuring gender bias in machine translation as the task of measuring gender errors in translation (as opposed to the prevalence of stereotyping in translation). We argue that oper- ationalizing the gender-bias measurement problem with an unambiguous task is much clearer than fram- ing it as an ambiguous task, because, in our setup, morphological gender mistakes are not forgivable. We introduce a novel unambiguous benchmark dataset that measures whether an MT model can appropriately inflect occupation nouns for gender when translating from an English source into 20 gender-marking target languages. We craft source sentences by manipulating the context of the occupation noun so that the gender of the person referred to (i.e., their gender identity) is clearly specified. For example: My nurse is a good father the gender identity of the nurse is unambiguous, because nurse is coreferent with father. When translating into a target, the occupation noun (nurse) requires …

Content truncated to 500 words