Enhancing Supervised Learning with Contrastive Markings in Neural Machine Translation Training

Enhancing Supervised Learning with Contrastive Markings in Neural Machine Translation Training Nathaniel Berger∗, Miriam Exel‡, Matthias Huck‡ and Stefan Riezler†,∗ ∗Computational Linguistics & †IWR, Heidelberg University, Germany ‡SAP SE, Dietmar-Hopp-Allee 16, 69190 Walldorf, Germany {berger, riezler}@cl.uni-heidelberg.de {miriam.exel, matthias.huck}@sap.com Abstract Supervised learning in Neural Machine Translation (NMT) typically follows a teacher forcing paradigm where reference tokens constitute the conditioning context in the model’s prediction, instead of its own previous predictions. In order to alleviate this lack of exploration in the space of translations, we present a sim- ple extension of standard maximum like- lihood estimation by a contrastive mark- ing objective. The additional training sig- nals are extracted automatically from ref- erence translations by comparing the sys- tem hypothesis against the reference, and used for up/down-weighting correct/incor- rect tokens. The proposed new training procedure requires one additional transla- tion pass over the training set per epoch, and does not alter the standard inference setup. We show that training with con- trastive markings yields improvements on top of supervised learning, and is espe- cially useful when learning from postedits where contrastive markings indicate hu- man error corrections to the original hy- potheses. Code is publicly released1. 1 Introduction Due to the availability of large parallel data sets for most language pairs, the standard training pro- cedure in Neural Machine Translation (NMT) is © 2023 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. 1https://www.cl.uni-heidelberg.de/ statnlpgroup/contrastive_markings/ supervised learning of a maximum likelihood ob- jective where reference tokens constitute the target history in the conditional language model, instead of the model’s own predictions. Feeding back the reference history in model training, known as teacher forcing (Williams and Zipser, 1989), en- courages the sequence model to stay close to the reference sequence, but prevents the model to learn how to predict conditioned on its own history, which is the actual task at inference time. This lack of exploration in learning has been dubbed exposure bias by Ranzato et al. (2016). It has been tackled by techniques that explicitly inte- grate the model’s own prediction history into train- ing, e.g. scheduled sampling (Bengio et al., 2015), minimum risk training (Shen et al., 2016), rein- forcement learning (Bahdanau et al., 2017), im- itation learning (Lin et al., 2020), or ramp loss (Jehl et al., 2019), amongst others. In most of these approaches, feedback from a human ex- pert is simulated by comparing a system transla- tion against a human reference according to an automatic evaluation metric, and by extracting a sequence- or token-level reward signal from the evaluation score. In this paper, we present a method to incorpo- rate contrastive markings of differences between the model’s own predictions and references into the learning objective. Our approach builds on pre- vious work on integrating weak human feedback in form of error markings as supervision signal in NMT training (Kreutzer et al., 2020). This work was conceptualized for reducing human annotation effort in interactive machine translation, however, it can also …

Content truncated to 500 words