The Effect of Round-Trip Translation on Fairness in Sentiment Analysis

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4423–4428 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4423 The Effect of Round-Trip Translation on Fairness in Sentiment Analysis Jonathan Gabel Christiansen∗ and Mathias Lykke Gammelgaard∗and Anders Søgaard Dpt. of Computer Science, University of Copenhagen Abstract Sentiment analysis systems have been shown to exhibit sensitivity to protected attributes. Round-trip translation, on the other hand, has been shown to normalize text. We explore the impact of round-trip translation on the de- mographic parity of sentiment classifiers and show how round-trip translation consistently improves classification fairness at test time (re- ducing up to 47% of between-group gaps). We also explore the idea of retraining sentiment classifiers on round-trip-translated data. 1 Introduction It is both unethical and potentially illegal for doc- ument classification algorithms to perform signif- icantly better for some groups in society than for others (Mehrabi et al., 2019). Many document clas- sification technologies have, however, been shown to be sensitive to protected attributes such as gen- der and age (Mehrabi et al., 2019; Delobelle et al., 2020; Ferrer et al., 2020; Koh et al., 2021). This also holds for sentiment analysis (Johannsen et al., 2015; Hovy, 2015; Kiritchenko and Mohammad, 2018; Bhaskaran and Bhallamudi, 2019; Touileb et al., 2020). At the same time it is known that round-trip machine translation (Huang, 1990; Fe- dermann et al., 2019) can be used to normalize text (Ling et al., 2013; Rabinovich et al., 2017; Prabhumoye et al., 2018). This could potentially remove group specific deviations from normal lan- guage. However, Stanovsky et al. (2019) found that machine translation is also prone to potentially introducing gender bias. Combined, this leaves it an open question whether round-trip translation can be used to reduce the sensitivity of document classifiers to protected attributes of the authors. In this paper, we evaluate the effect of round- trip translation on fairness using a representative ∗Equal contributions. Corresponding author: soegaard@di.ku.dk. Text Sally is a whiz at math. Sally es una experta en matem´aticas. Sally is a math expert. Table 1: The normalizing effect of round-trip transla- tion: translating English to Spanish and back. corpus of Danish Trustpilot reviews, in which re- views are associated with self-reported protected attributes (gender and age) (Hovy et al., 2015). We evaluate this effect across nine different document classification architectures, both in the setting in which round-trip translation happens at test time only, and in the setting in which both training and test data are translated to a foreign language and back. Contributions We evaluate round-trip transla- tion as a technique for mitigating sensitivity to protected attributes across two attributes and three document classification architectures. We find that round-trip translation at test time consistently re- duces the fairness gap (with up to 47%), but that for our best models (SVMs stacked on BERT represen- tations), the effect disappears when both training and test data are translated into a foreign language and back. 2 Round-trip Translation Round-trip machine translation (Huang, 1990; Fe- …

Content truncated to 500 words