A Reinforcement Learning Approach to Interactive-Predictive Neural Machine Translation

A Reinforcement Learning Approach to Interactive-Predictive Neural Machine Translation Tsz Kin Lam†,∗ and Julia Kreutzer∗ and Stefan Riezler†,∗ ∗Computational Linguistics & †IWR, Heidelberg University, Germany {lam,kreutzer,riezler}@cl.uni-heidelberg.de Abstract We present an approach to interactive- predictive neural machine translation that attempts to reduce human effort from three directions: Firstly, instead of requiring hu- mans to select, correct, or delete segments, we employ the idea of learning from hu- man reinforcements in form of judgments on the quality of partial translations. Sec- ondly, human effort is further reduced by using the entropy of word predictions as uncertainty criterion to trigger feedback requests. Lastly, online updates of the model parameters after every interaction allow the model to adapt quickly. We show in simulation experiments that re- ward signals on partial translations sig- nificantly improve character F-score and BLEU compared to feedback on full trans- lations only, while human effort can be re- duced to an average number of 5 feedback requests for every input. 1 Introduction Interactive-predictive machine translation aims at obtaining high-quality machine translation by in- volving humans in a loop of user validations of partial translations suggested by the machine translation system. This interaction protocol can easily be fit to neural machine translation (NMT) (Bahdanau et al., 2015) by conditioning the model’s word predictions on the user-validated prefix (Knowles and Koehn, 2016; Wuebker et al., 2016). User studies conducted by Green et c⃝ 2018 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. al. (2014) for phrase-based machine translation have shown that the interactive-predictive inter- action protocol leads to significant reductions in post-editing effort. Other user studies on interac- tive machine translation based on post-editing have shown that human effort can also be reduced by improving the online adaptation capabilities of the learning system, both for statistical phrase-based (Bentivogli et al., 2016) or NMT systems (Kari- mova et al., 2017). The goal of our work is to further reduce human effort in interactive-predictive NMT by combining the advantages of the interactive-predictive pro- tocol with the advantages of learning from weak feedback. For the latter we rely on techniques from reinforcement learning (Sutton and Barto, 2017), a.k.a. bandit structured prediction (Sokolov et al., 2016; Kreutzer et al., 2017; Nguyen et al., 2017) in the context of sequence-to-sequence learning. Our approach attacks the problem of reducing human effort from three innovative directions. • Firstly, instead of requiring humans to cor- rect or delete segments proposed by the ma- chine translation system, we employ the re- inforcement learning idea of humans provid- ing reward signals in form of judgments on the quality of the machine translation. Hu- man effort is reduced since each partial trans- lation receives a human reward signal at most once, rendering it a bandit-type feedback sig- nal, and each reward signal itself is easier to obtain than a correction of a translation. • In order to reduce the amount of feedback signals even further, we integrate an uncer- tainty criterion for word …

Content truncated to 500 words