Contrastive Conditioning for Assessing Disambiguation in MT: A Case Study of Distilled Bias

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10246–10265 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10246 Contrastive Conditioning for Assessing Disambiguation in MT: A Case Study of Distilled Bias Jannis Vamvas1 and Rico Sennrich1,2 1Department of Computational Linguistics, University of Zurich 2School of Informatics, University of Edinburgh {vamvas,sennrich}@cl.uzh.ch Abstract Lexical disambiguation is a major challenge for machine translation systems, especially if some senses of a word are trained less often than others. Identifying patterns of overgener- alization requires evaluation methods that are both reliable and scalable. We propose con- trastive conditioning as a reference-free black- box method for detecting disambiguation er- rors. Specifically, we score the quality of a translation by conditioning on variants of the source that provide contrastive disambiguation cues. After validating our method, we apply it in a case study to perform a targeted evaluation of sequence-level knowledge distillation. By probing word sense disambiguation and trans- lation of gendered occupation names, we show that distillation-trained models tend to over- generalize more than other models with a com- parable BLEU score. Contrastive condition- ing thus highlights a side effect of distillation that is not fully captured by standard evalua- tion metrics. Code and data to reproduce our findings are publicly available.1 1 Introduction Erroneous disambiguation of words makes trans- lations inadequate and can even constitute a form of bias when it occurs systematically. However, detecting disambiguation errors in machine transla- tion (MT) is a non-trivial task. Previous work has focused on automatic post-hoc analysis of transla- tions (Raganato et al., 2019; Stanovsky et al., 2019), but rules of what makes a disambiguation correct or incorrect tend to be imprecise. While contrastive evaluation (Sennrich, 2017; Rios et al., 2017) elim- inates the need for post-hoc analysis by scoring pre-defined pairs of hypotheses, such probability estimates cannot be obtained from black-box sys- tems, e.g., from commercial APIs that only return 1https://github.com/ZurichNLP/ contrastive-conditioning doctor Original corpus Teacher Student 2.0% 77.8% 1.4% 83.5% 1.1% 85.1% Figure 1: Our case study is motivated by an analysis of the training data: In the English–German WMT19 news corpus, doctor is mostly translated into male forms such as Arzt and rarely into female forms (center: other variants). Student models trained on a machine- translated version of the data amplify this imbalance. a 1-best translation to the user. In addition, con- trastive hypotheses need to be carefully crafted for every target language of interest. We propose contrastive conditioning as a scal- able black-box alternative for evaluating disam- biguation in machine translation. The evaluated translations are paired with contrastive source se- quences and are then scored by a white-box trans- lation model. The contrastive sources are variants of the original source, slightly modified to provide a stronger disambiguation cue. For example, con- sider a model that translates the English source ‘doctor’ into German as ‘Ärztin’ (female doctor). This translation will receive a better score when conditioned on the source ‘female doctor’ than on ‘male doctor’, indicating that it is a feminine …

Content truncated to 500 words