Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1673–1681 June 6–11, 2021. ©2021 Association for Computational Linguistics 1673 Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation Samuel Kiegeland Heidelberg University kiegeland@cl.uni-heidelberg.de Julia Kreutzer Google Research jkreutzer@google.com Abstract Policy gradient algorithms have found wide adoption in NLP, but have recently become subject to criticism, doubting their suitability for NMT. Choshen et al. (2020) identify multi- ple weaknesses and suspect that their success is determined by the shape of output distribu- tions rather than the reward. In this paper, we revisit these claims and study them under a wider range of configurations. Our experi- ments on in-domain and cross-domain adapta- tion reveal the importance of exploration and reward scaling, and provide empirical counter- evidence to these claims. 1 Introduction In neural sequence-to-sequence learning, in particu- lar Neural Machine Translation (NMT), Reinforce- ment Learning (RL) has gained attraction due to the suitability of Policy Gradient (PG) methods for the end-to-end training paradigm (Ranzato et al., 2016; Li et al., 2016; Yu et al., 2017; Li et al., 2018; Flachs et al., 2019; Sankar and Ravi, 2019). The idea is to let the model explore the output space be- yond the reference output that is used for standard cross-entropy minimization, by reinforcing model outputs according to their quality, effectively in- creasing the likelihood of higher-quality samples. The classic exploration-exploitation dilemma from RL is addressed by sampling from a pretrained model’s softmax distribution over output tokens, such that the model entropy steers exploration. For the application of NMT, it was firstly uti- lized to bridge the mismatch between the optimiza- tion for token-level likelihoods during training and the corpus-level held-out set evaluations with non- differentiable/decomposable metrics like BLEU (Ranzato et al., 2016; Edunov et al., 2018), and secondly to reduce exposure bias in autoregressive sequence generators (Ranzato et al., 2016; Wang and Sennrich, 2020). It has furthermore been identi- fied as a promising tool to adapt pretrained models to new domains or user preferences by replacing reward functions with human feedback in human- in-the-loop learning (Sokolov et al., 2016; Nguyen et al., 2017). Recently, the effectiveness of these methods has been questioned: Choshen et al. (2020) iden- tify multiple theoretical and empirical weaknesses, leading to the suspicion that performance gains with RL in NMT are not due to the reward signal. The most surprising result is that the replacement of a meaningful reward function (giving higher re- wards to higher-quality translations) by a constant reward (reinforcing all model samples equally) yields similar improvements in BLEU. To explain this counter-intuitive result, Choshen et al. (2020) conclude that a phenomenon called the peakiness effect must be responsible for performance gains instead of the reward. This means that the most likely tokens in the beginning gain probability mass regardless of the rewards they receive during RL training. If this hypothesis was true, then the per- spectives for using methods of RL for encoding real-world …

Content truncated to 500 words