Interactive-Predictive Neural Machine Translation through Reinforcement and Imitation

Interactive-Predictive Neural Machine Translation through Reinforcement and Imitation Tsz Kin Lam∗ and Shigehiko Schamoni†,∗ and Stefan Riezler†,∗ ∗Computational Linguistics & †IWR, Heidelberg University, Germany {lam,schamoni,riezler}@cl.uni-heidelberg.de Abstract We propose an interactive-predictive neu- ral machine translation framework for eas- ier model personalization using reinforce- ment and imitation learning. During the interactive translation process, the user is asked for feedback on uncertain locations identified by the system. Responses are weak feedback in the form of “keep” and “delete” edits, and expert demonstrations in the form of “substitute” edits. Condi- tioning on the collected feedback, the sys- tem creates alternative translations via con- strained beam search. In simulation exper- iments on two language pairs our systems get close to the performance of supervised training with much less human effort. 1 Introduction Despite recent success reports on neural machine translation (NMT) reaching human parity (Wu et al., 2016; Hassan et al., 2018), professional use cases of NMT require model personalization where the NMT system is adapted to user feedback provided for suggested NMT outputs (Wuebker et al., 2018; Michel and Neubig, 2018). In this pa- per, we will focus on the paradigm of interactive- predictive machine translation (Foster et al., 1997; Barrachina et al., 2008) which has been shown to fit easily into the sequence-to-sequence model of NMT (Knowles and Koehn, 2016; Wuebker et al., 2016). The standard interactive-predictive proto- col takes a human-corrected prefix as condition- ing context in predicting a sentence completion, c⃝ 2019 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. which is again corrected or accepted by the human user. Recent work showed in simulation experi- ments that human effort can be reduced by asking humans for reward signals or validations of partial system outputs instead of for corrections (Lam et al., 2018; Domingo et al., 2017). Our goal is to combine both feedback modes — corrections and rewards — by treating them as expert demonstrations and reward values in an in- teractive protocol that combines imitation learning (IL) (Ross et al., 2011) and reinforcement learning (RL) (Sutton and Barto, 2018), respectively, us- ing only limited human edits. A further difference of our framework to standard interactive-predictive NMT is our use of an uncertainty criterion that re- duces the amount of feedback requests to the to- kens where the entropy of the policy distribution is highest. This idea has been used successfully before in Lam et al. (2018) and Peris and Casacu- berta (2018) and connects our work to the area of active learning (Settles and Craven, 2008). Lastly, our framework differs from prior work by allowing model updates based on partial translations. Our experiments show that weak feedback in form of keep/delete rewards on translation outputs yields consistent improvements of between 2.6 and 4.3 BLEU points over the pre-trained baseline. On one language pair, it even matches the improve- ments gained by forcing word substitutions from reference translations into the re-decoded output. Furthermore, both feedback scenarios consider- ably reduce human …

Content truncated to 500 words