Exploring Supervised and Unsupervised Rewards in Machine Translation

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1908–1920 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1908 Exploring Supervised and Unsupervised Rewards in Machine Translation Julia Ive1, Zixu Wang1, Marina Fomicheva2, Lucia Specia1,2,3 Imperial College London1, University of Sheffield2, ADAPT - Dublin City University3 j.ive@ic.ac.uk, zixu.wang@imperial.ac.uk m.fomicheva@sheffield.ac.uk, l.specia@ic.ac.uk Abstract Reinforcement Learning (RL) is a powerful framework to address the discrepancy between loss functions used during training and the fi- nal evaluation metrics to be used at test time. When applied to neural Machine Translation (MT), it minimises the mismatch between the cross-entropy loss and non-differentiable eval- uation metrics like BLEU. However, the suit- ability of these metrics as reward function at training time is questionable: they tend to be sparse and biased towards the specific words used in the reference texts. We propose to ad- dress this problem by making models less re- liant on such metrics in two ways: (a) with an entropy-regularised RL method that does not only maximise a reward function but also ex- plore the action space to avoid peaky distri- butions; (b) with a novel RL method that ex- plores a dynamic unsupervised reward func- tion to balance between exploration and ex- ploitation. We base our proposals on the Soft Actor-Critic (SAC) framework, adapting the off-policy maximum entropy model for language generation applications such as MT. We demonstrate that SAC with BLEU reward tends to overfit less to the training data and performs better on out-of-domain data. We also show that our dynamic unsupervised re- ward can lead to better translation of ambigu- ous words. 1 Introduction Autoregressive sequence-to-sequence (seq2seq) neural architectures have become the de facto approach in Machine Translation (MT). Such models include Recurrent Neural Networks (RNN) (Sutskever et al., 2014; Bahdanau et al., 2014) and Transformer networks (Vaswani et al., 2017), among others. However, these models have as a serious limitation the discrepancy between their training and inference time regimes. They are traditionally trained using the Maximum Like- lihood Estimation (MLE), which aims to maximise log-likelihood of a categorical ground truth distri- bution (samples in the training corpus) using loss functions such as cross-entropy, which are very dif- ferent from the evaluation metric used at inference time, which generally compares string similarity between the system output and reference outputs. Moreover, during training, the generator receives the ground truth as input and is trained to minimise the loss of a single token at a time without taking the sequential nature of language into account. At inference time, however, the generator will take the previous sampled output as the input at next time step, rather than the ground truth word. MLE train- ing thus causes: (a) the problem of “exposure bias” as a result of recursive conditioning on its own er- rors at test time, since the model has never been exclusively “exposed” to its own predictions during training; (b) a mismatch between the training ob- jective and the test objective, where the …

Content truncated to 500 words