SURF: Semantic-level Unsupervised Reward Function for Machine Translation

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4508 - 4522 July 10-15, 2022 ©2022 Association for Computational Linguistics SURF: Semantic-level Unsupervised Reward Function for Machine Translation Atijit Anuchitanukul1, Julia Ive2 Imperial College London1, Queen Mary University of London2 atijit.anuchitanukul20@imperial.ac.uk j.ive@qmul.ac.uk Abstract The performance of Reinforcement Learning (RL) for natural language tasks including Ma- chine Translation (MT) is crucially depen- dent on the reward formulation. This is due to the intrinsic difficulty of the task in the high-dimensional discrete action space as well as the sparseness of the standard reward functions defined for limited set of ground- truth sequences biased towards singular lexi- cal choices. To address this issue, we formu- late SURF, a maximally dense semantic-level unsupervised reward function which mimics human evaluation by considering both sen- tence fluency and semantic similarity. We demonstrate the strong potential of SURF to leverage a family of Actor-Critic Transformer- based Architectures with synchronous and asynchronous multi-agent variants. To tackle the problem of large action-state spaces, each agent is equipped with unique exploration strategies, promoting diversity during its ex- ploration of the hypothesis space. When BLEU scores are compared, our dense un- supervised reward outperforms the standard sparse reward by 2% on average for in- and out-of-domain settings. 1 Introduction Reinforcement Learning (RL) has shown promise in the field of text generation. This is mainly due to the fact that it allows the usage of non- differentiable evaluation functions fit for the dis- crete natural language tasks. It also serves as a solution for bridging the gap between training and inference time regimes (“exposure bias”) that arises from the fact that the model is never exposed to its own errors as only ground-truth labels are used to condition the generation during training (Wang and Sennrich, 2020). One of the essential components of the RL framework is the reward function, which is used to provide agents with indicative signals in terms of the effectiveness of chosen actions. The usage of RL in Neural Machine Transla- tion (NMT) and language generation however has been doubted largely due to the difficulty of ex- ploration in the high-dimensional discrete action space combined with the sparse reward signal. The latter comes from the typical metrics used as re- wards (e.g., BLEU (Papineni et al., 2002)). These rewards evaluate text in a shallow way by mea- suring the string similarity between generated and ground-truth sequences, making them extremely sparse and biased towards singular lexical choices. As the RL policy is usually initialised with some pre-trained distribution over words, suspicion has been raised that in this situation, those words al- ready most likely gain probability mass regardless of the rewards (Choshen et al., 2020). Thus, the current sparse rewards are not beneficial for rigor- ous exploration of different words during training. Recent studies suggest that the main benefit for NMT from RL is in performing domain adaptation when using proper hypothesis space exploration along with special emphasis on …

Content truncated to 500 words