Bias Mitigation in Machine Translation Quality Estimation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1475 - 1487 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Bias Mitigation in Machine Translation Quality Estimation Hanna Behnke Imperial College London hanna.behnke20@ic.ac.uk Marina Fomicheva University of Sheffield m.fomicheva@sheffield.ac.uk Lucia Specia Imperial College London l.specia@ic.ac.uk Abstract Machine Translation Quality Estimation (QE) aims to build predictive models to assess the quality of machine-generated translations in the absence of reference translations. While state-of-the-art QE models have been shown to achieve good results, they over-rely on fea- tures that do not have a causal impact on the quality of a translation. In particular, there appears to be a partial input bias, i.e., a ten- dency to assign high-quality scores to transla- tions that are fluent and grammatically correct, even though they do not preserve the meaning of the source. We analyse the partial input bias in further detail and evaluate four approaches to use auxiliary tasks for bias mitigation. Two approaches use additional data to inform and support the main task, while the other two are adversarial, actively discouraging the model from learning the bias. We compare the meth- ods with respect to their ability to reduce the partial input bias while maintaining the overall performance. We find that training a multitask architecture with an auxiliary binary classifi- cation task that utilises additional augmented data best achieves the desired effects and gen- eralises well to different languages and quality metrics. 1 Introduction Despite the great advances of Machine Translation (MT) models over the past years, the adequacy and fluency of the translations cannot be guaranteed. Without access to a gold-standard reference transla- tion, it can be difficult to validate the reliability of the MT model’s predictions. To address this issue, the field of MT Quality Estimation (QE) emerged, aiming to develop models that can approximate the quality of machine-generated translations in a scal- able way. However, recent research suggests that state-of-the-art QE approaches tend to over-rely on features that do not have a causal impact on the quality of a translation. In particular, there appears to be a partial input bias, i.e. a tendency to assign high quality scores to translations that are fluent and grammatical, even though they do not resemble the actual meaning of the source (Sun et al., 2020). Building upon these findings, the objective of our work is to characterise and, most importantly, mitigate the partial input bias of QE models. We focus on the use of auxiliary training tasks to specifically target the observed biases while avoid- ing strong modifications of the original model as well as the expensive collection and manual la- belling of additional training data. Our efforts concentrate on testing and improving MonoTran- sQuest, the best-performing architecture in the shared task on sentence-level QE hosted as part of the Fifth Conference on Machine Translation (WMT 2020) (Specia et al., 2020). We work with the recently published multilingual QE dataset MLQE-PE (Fomicheva et al., 2020), allowing us to …

Content truncated to 500 words