Gender Coreference and Bias Evaluation at WMT 2020

Proceedings of the 5th Conference on Machine Translation (WMT), pages 357–364 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 357 Gender Coreference and Bias Evaluation at WMT 2020 Tom Kocmi ∗ Microsoft tomkocmi@microsoft.com Tomasz Limisiewicz Charles University in Prague Faculty of Mathematics and Physics limisiewicz@ufal.mff.cuni.cz Gabriel Stanovsky The Hebrew University of Jerusalem gabis@cse.huji.ac.il Abstract Gender bias in machine translation can man- ifest when choosing gender inflections based on spurious gender correlations. For exam- ple, always translating doctors as men and nurses as women. This can be particularly harmful as models become more popular and deployed within commercial systems. Our work presents the largest evidence for the phe- nomenon in more than 19 systems submit- ted to the WMT over four diverse target lan- guages: Czech, German, Polish, and Russian. To achieve this, we use WinoMT, a recent auto- matic test suite which examines gender coref- erence and bias when translating from English to languages with grammatical gender. We ex- tend WinoMT to handle two new languages tested in WMT: Polish and Czech. We find that all systems consistently use spurious correla- tions in the data rather than meaningful con- textual information. 1 Introduction Bias in machine learning occurs when systems pick up correlations which are useful for specific train- ing datasets, but are not indicative for the task that the dataset represents. In the context of machine translation (MT), gen- der bias can occur when translating from languages without grammatical noun genders, such as English or Turkish, to a language with gender inflections, such as Spanish, Polish, or Czech. In such cases, the translation model needs to assign gender in- flection in the target language based on contextual cues in the source text. For example, when trans- lating the English sentence “The doctor asked the nurse to help her in the operation” to Spanish, the model needs to produce the female inflected “doc- tora” based on the feminine English pronoun “her”. ∗ Part of work performed while at Charles University. Recently, Stanovsky et al. (2019) created a challenge set and an automatic evaluation metric, dubbed WinoMT, to examine whether popular MT models are capable of correctly capturing and trans- lating such information from English into a diverse set of 8 target languages with grammatical gender. They found that all six tested systems, composed of four commercial and two academic models, con- sistently relied on gender role assignments in the data regardless of context. In our example above, models would prefer to translate the doctor using masculine inflections, despite the context suggest- ing otherwise. In this work, we apply the WinoMT test suite on the submissions to the News shared task of WMT 2020. In addition to testing the phenomenon on a large number of models, we extend the WinoMT to the Polish and Czech languages, tackling unique language-specific challenges. We thoroughly ana- lyze the extent of the phenomena for the tested lan- guages and systems, as well as its correlation with the widely-used BLEU evaluation metric (Papineni et al., …

Content truncated to 500 words