“You Sound Just Like Your Father” Commercial Machine Translation Systems Include Stylistic Biases

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1686–1690 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1686 “You Sound Just Like Your Father” Commercial Machine Translation Systems Include Stylistic Biases Dirk Hovy Federico Bianchi Bocconi University Via Sarfatti 25, 20136 Milan, Italy {dirk.hovy, f.bianchi, fornaciari.tommaso}@unibocconi.it Tommaso Fornaciari Abstract The main goal of machine translation has been to convey the correct content. Stylistic con- siderations have been at best secondary. We show that as a consequence, the output of three commercial machine translation systems (Bing, DeepL, Google) make demographically diverse samples from five languages “sound” older and more male than the original. Our findings suggest that translation models reflect demographic bias in the training data. These results open up interesting new research av- enues in machine translation to take stylistic considerations into account. 1 Introduction Translating what is being said is arguably the most important aspect of machine translation, and has been the main focus of all its efforts so far. How- ever, how something is said also has an impact on how the final translation is perceived. Mirkin et al. (2015) have pointed out that demographic as- pects of language do play a role in translation, and could help in personalization. As Vanmassenhove et al. (2018) have shown, gendered inflections like “Sono stanco/a” (Italian I am tired) are an impor- tant aspect of correct translations. In many cases, capturing the style of a docu- ment is equally important as its content: translat- ing a lover’s greeting as “I am entirely pleased to see you” might be semantically correct, but seems out of place. Demographic factors (age, gender, etc.) all manifest in language, and therefore influ- ence style: we do not expect a 6-year old to sound like an adult, and would not translate a person to seem differently gendered. However, in this paper, we show such a change is essentially what happens in machine translation: authors sound on average older and more male. Prior work (Rabinovich et al., 2017) has shown that translation weakens the signal for gender pre- diction. We substantially extend this analysis in terms of languages, demographic factors, and types of models, controlling for demographically representative samples. We show the direction in which the predicted demographic factors differ in the translations, and find that there are consis- tent biases towards older and more male profiles. Our findings suggest a severe case of overexpo- sure to writings from these demographics (Hovy and Spruit, 2016), which creates a self-reinforcing loop. In this paper, we use demographically- representative author samples from five languages (Dutch, English, French, German, Italian), and translate them with three commercially available machine translation systems (Google, Bing, and DeepL). We compare the true demographics with the predicted demographics of each translation (as well as a control predictor trained on the same language). Without making any judgment on the translation of the content, we find a) that there are substantial discrepancies in the perceived demographics, and b) that translations tend …

Content truncated to 500 words