First the Worst: Finding Better Gender Translations During Beam Search

Findings of the Association for Computational Linguistics: ACL 2022, pages 3814 - 3823 May 22-27, 2022 c⃝2022 Association for Computational Linguistics First the Worst: Finding Better Gender Translations During Beam Search Danielle Saunders∗and Rosie Sallis and Bill Byrne Department of Engineering, University of Cambridge, UK {ds636, rs965}@cantab.ac.uk, wjb31@cam.ac.uk Abstract Generating machine translations via beam search seeks the most likely output under a model. However, beam search has been shown to amplify demographic biases exhibited by a model. We aim to address this, focusing on gender bias resulting from systematic errors in grammatical gender translation. Almost all prior work on this problem adjusts the training data or the model itself. By contrast, our ap- proach changes only the inference procedure. We constrain beam search to improve gender diversity in n-best lists, and rerank n-best lists using gender features obtained from the source sentence. Combining these strongly improves WinoMT gender translation accuracy for three language pairs without additional bilingual data or retraining. We also demonstrate our approach’s utility for consistently gendering named entities, and its flexibility to handle new gendered language beyond the binary. 1 Introduction Neural language generation models optimized by likelihood have a tendency towards ‘safe’ word choice. This lack of output diversity has been noted in NMT (Vanmassenhove et al., 2019) and throughout NLP (Li et al., 2016; Sultan et al., 2020). Model-generated language may be repetitive or stilted. More insidiously, generating the most likely output based only on corpus statistics can amplify any existing biases in the corpus (Zhao et al., 2017). Potential harms arise when biases around word choice or grammatical gender inflections reflect de- mographic or social biases (Sun et al., 2019). The resulting gender mistranslations could involve im- plicit misgendering of a user or other referent, or perpetuation of social stereotypes about the ‘typi- cal’ gender of a referent in a given context. Past approaches to the problem almost exclu- sively involve retraining (Vanmassenhove et al., ∗Now at RWS Language Weaver 2018; Escudé Font and Costa-jussà, 2019; Stafanoviˇcs et al., 2020) or tuning (Saunders and Byrne, 2020; Basta et al., 2020) on gender-adjusted data. Such approaches are often computationally expensive and risk introducing new biases (Shah et al., 2020). Instead, we seek to improve transla- tions from existing models. Roberts et al. (2020) highlight beam search’s tendency to amplify gender bias and Renduchintala et al. (2021) show that very shallow beams degrade gender translation accu- racy; we instead guide beam search towards better gender translations further down the n-best list. Our contributions are as follows: we rerank NMT n-best lists, demonstrating that we can extract better gender translations from the original model’s beam. We also generate new n-best lists subject to gendered inflection constraints, and show this makes correctly gendered entities more common in n-best lists. We make no changes to the NMT model or training data, and require only monolin- gual resources for the source and target languages. 1.1 Related work Prior work mitigating gender bias in NLP often in- volves adjusting training data, …

Content truncated to 500 words