Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1464–1474 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback Khanh Nguyen⊙♦ and Hal Daum´e III⊙♠♦♥ and Jordan Boyd-Graber⊙♠♣♦ University of Maryland: Computer Science,⊙Language Science,♠iSchool,♣UMIACS♦ Microsoft Research, New York♥ {kxnguyen,hal,jbg}@umiacs.umd.edu Abstract Machine translation is a natural candidate problem for reinforcement learning from human feedback: users provide quick, dirty ratings on candidate translations to guide a system to improve. Yet, current neural machine translation training fo- cuses on expensive human-generated ref- erence translations. We describe a re- inforcement learning algorithm that im- proves neural machine translation sys- tems from simulated human feedback. Our algorithm combines the advantage actor-critic algorithm (Mnih et al., 2016) with the attention-based neural encoder- decoder architecture (Luong et al., 2015). This algorithm (a) is well-designed for problems with a large action space and delayed rewards, (b) effectively optimizes traditional corpus-level machine transla- tion metrics, and (c) is robust to skewed, high-variance, granular feedback modeled after actual human behaviors. 1 Introduction Bandit structured prediction is the task of learning to solve complex joint prediction problems (like parsing or machine translation) under a very lim- ited feedback model: a system must produce a sin- gle structured output (e.g., translation) and then the world reveals a score that measures how good or bad that output is, but provides neither a “cor- rect” output nor feedback on any other possible output (Chang et al., 2015; Sokolov et al., 2015). Because of the extreme sparsity of this feedback, a common experimental setup is that one pre-trains a good-but-not-great “reference” system based on whatever labeled data is available, and then seeks to improve it over time using this bandit feedback. A common motivation for this problem setting is cost. In the case of translation, bilingual “ex- perts” can read a source sentence and a possible translation, and can much more quickly provide a rating of that translation than they can produce a full translation on their own. Furthermore, one can often collect even less expensive ratings from “non-experts” who may or may not be bilingual (Hu et al., 2014). Breaking this reliance on ex- pensive data could unlock previously ignored lan- guages and speed development of broad-coverage machine translation systems. All work on bandit structured prediction we know makes an important simplifying assumption: the score provided by the world is exactly the score the system must optimize (§2). In the case of pars- ing, the score is attachment score; in the case of machine translation, the score is (sentence-level) BLEU. While this simplifying assumption has been incredibly useful in building algorithms, it is highly unrealistic. Any time we want to optimize a system by collecting user feedback, we must take into account: 1. The metric we care about (e.g., expert ratings) may not correlate perfectly with the measure that the reference system was trained on (e.g., BLEU or log likelihood); 2. Human judgments might be more granu- lar …

Content truncated to 500 words