MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4287 - 4299 December 7-11, 2022 ©2022 Association for Computational Linguistics MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation Anna Currey, Maria N˘adejde, Raghavendra Pappagari, Mia Mayer, Stanislas Lauly, Xing Niu, Benjamin Hsu, Georgiana Dinu AWS AI Labs ancurrey@amazon.com Abstract As generic machine translation (MT) quality has improved, the need for targeted bench- marks that explore fine-grained aspects of qual- ity has increased (Freitag et al., 2021; Isabelle et al., 2017). In particular, gender accuracy in translation (Choubey et al., 2021; Saunders and Byrne, 2020) can have implications in terms of output fluency, translation accuracy, and ethics. In this paper, we introduce MT- GenEval, a benchmark for evaluating gender accuracy in translation from English into eight widely-spoken languages. MT-GenEval com- plements existing benchmarks by providing re- alistic, gender-balanced, counterfactual data in eight language pairs where the gender of indi- viduals is unambiguous in the input segment, including multi-sentence segments requiring inter-sentential gender agreement. Our data and code is publicly available under a CC BY SA 3.0 license.1 1 Introduction Although neural machine translation (NMT) has made great strides in quality (Hassan et al., 2018; Wu et al., 2016), evaluations on generic test sets may not tell the whole story. Indeed, NMT models are known to make systematic errors in areas like robustness to input perturbations (Niu et al., 2020), disambiguating pronouns in context (Müller et al., 2018), and translating unambiguous human gen- der (Stanovsky et al., 2019). In particular, gender- related issues in translation can lead to translations that are inaccurate, ungrammatical, or biased. Accordingly, there has been increasing inter- est in improving machine translation for gendered entities (Bentivogli et al., 2020; Choubey et al., 2021; Saunders and Byrne, 2020; Savoldi et al., 2021; Stanovsky et al., 2019). Adequate evaluation benchmarks play an important role in supporting 1https://github.com/amazon-research/ machine-translation-gender-eval this line of research and improving understanding of how models perform on the task of gender trans- lation accuracy. Existing benchmarks have limited diversity in terms of gender phenomena (e.g., focus- ing on professions), sentence structure (e.g., using templates to construct sentences), or language cov- erage (see section 4 for more information), making it difficult to gauge how systems perform in terms of both gender and quality simultaneously. To this end, this paper releases a new Machine Translation Gender Evaluation benchmark: MT- GenEval. We include development data (for model improvements) as well as test data and correspond- ing metrics (for comprehensive evaluation). MT- GenEval is realistic and diverse, includes a wide variety of contexts for gender disambiguation, and is fully balanced by including human-created gen- der counterfactuals. In its first release, the MT- GenEval benchmark covers translation from En- glish into eight target languages, for two genders. MT-GenEval focuses on the task of gender accu- racy in translation. We define gender accuracy in translation as the extent to which a machine trans- lation output accurately reflects the gender of the humans mentioned in …

Content truncated to 500 words