Extending Challenge Sets to Uncover Gender Bias in Machine Translation: Impact of Stereotypical Verbs and Adjectives

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 531–541 November 10–11, 2021. ©2021 Association for Computational Linguistics 531 Extending Challenge Sets to Uncover Gender Bias in Machine Translation Impact of Stereotypical Verbs and Adjectives Jonas Troles Cognitive Systems University of Bamberg jonas.troles@uni-bamberg.de Ute Schmid Cognitive Systems University of Bamberg ute.schmid@uni-bamberg.de Abstract Human gender bias is reflected in language and text production. Because state-of-the-art machine translation (MT) systems are trained on large corpora of text, mostly generated by humans, gender bias can also be found in MT. For instance when occupations are translated from a language like English, which mostly uses gender neutral words, to a language like German, which mostly uses a feminine and a masculine version for an occupation, a deci- sion must be made by the MT System. Re- cent research showed that MT systems are bi- ased towards stereotypical translation of occu- pations. In 2019 the first, and so far only, chal- lenge set, explicitly designed to measure the extent of gender bias in MT systems has been published. In this set measurement of gender bias is solely based on the translation of oc- cupations. With our paper we present an ex- tension of this challenge set, called WiBeMT1, which adds gender-biased adjectives and sen- tences with gender-biased verbs. The result- ing challenge set consists of over 70, 000 sen- tences and has been translated with three com- mercial MT systems: DeepL Translator, Mi- crosoft Translator, and Google Translate. Re- sults show a gender bias for all three MT sys- tems. This gender bias is to a great extent significantly influenced by adjectives and to a lesser extent by verbs. 1 Introduction The problem of unfair and biased models has been recognized as an important problem for many appli- cations of machine learning (Mehrabi et al., 2019). The source of unfairness typically is based on bi- ases in the training data. In many domains, un- fairness is caused by sampling biases. In machine translation (MT), however, the main source of un- fairness is due to historical or social biases when there is a misalignment between the world as it 1Our test set and related data is available at: www. github.com/JDtroles/WiBeMTdata.git is and the values or objectives to be encoded and propagated in a model (Suresh and Guttag, 2019; Bolukbasi et al., 2016). One source of imbalance in natural language is the association of specific occupations with gender. Typically, occupations in more technical domains as well as occupations with high social status are associated with male gender (Cheryan et al., 2017). In natural language processing (NLP), gender bias has been investigated for word embeddings (Bolukbasi et al., 2016). Analogy puzzles such as “man is to king as woman is to x” generated with word2vec2 yield x = queen while for “man is to computer programmer as woman is to x”, the output is x = homemaker. Only few publica- tions exist that directly address gender bias and MT. Although gender bias is particularly relevant for translations …

Content truncated to 500 words