Automatically Identifying Gender Issues in Machine Translation using Perturbations

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1991–1995 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1991 Automatically Identifying Gender Issues in Machine Translation using Perturbations Hila Gonen Bar-Ilan University hilagnn@gmail.com Kellie Webster Google Research NYC websterk@google.com Abstract The successful application of neural methods to machine translation has realized huge quality advances for the community. With these improvements, many have noted outstanding challenges, including the modeling and treatment of gendered language. While previous studies have identified issues using synthetic examples, we develop a novel technique to mine examples from real world data to explore challenges for deployed systems. We use our method to compile an evaluation benchmark spanning examples for four languages from three language families, which we publicly release to facilitate research. The examples in our benchmark expose where model representations are gendered, and the unintended consequences these gendered representations can have in downstream application. 1 Introduction Machine translation (MT) has realized huge improvements in quality from the successful application and development of neural methods (Kalchbrenner and Blunsom, 2013; Cho et al., 2014; Vaswani et al., 2017; Johnson et al., 2017; Chen et al., 2018). As the community has explored this enhanced performance, many have noted the outstanding challenge of modeling and handling gendered language (Kuczmarski, 2018; Escud´e Font and Costa-juss`a, 2019). We extend this line of work, which identifies issues using synthetic examples manually curated for a target language (Stanovsky et al., 2019; Cho et al., 2019), by analyzing real world text across a range of languages to understand challenges for deployed systems. In this paper, we explore the class of issues which surface when a neutral reference to a person is translated to a gendered form (e.g. in Table 1, where the English counselor and nurse are translated into the French conseiller (masculine) and infirmi`ere (feminine). For this class of examples, the MT task requires a system to produce a single translation without source cues, thus exposing a model’s preferred gender for the reference form. With this scope, we make two key contributions. First, we design and implement an automatic pipeline for detecting examples of our class of gender issues in real world input, using a BERT- based perturbation method novel to this work. A key advantage of our pipeline beyond previous work is its extensibility: a) beyond word lists; b) to different language pairs and c) parts of speech. Second, using our new pipeline, we compile a dataset that we make publicly available to serve as a benchmark for future work. We focus on English as the source language, and explore four target gendered languages across three language families (French, German, Spanish, and Russian). Our examples expose where MT encodings are gendered, finding new issues not covered in previous manual approaches, and the unintended consequences of this for translation. 2 Gender Marking Languages Gender-marking languages have rich grammatical systems for expressing gender (Corbett, 1991). To produce a valid sentence in a gender-marking language, gender may need to be marked not only …

Content truncated to 500 words