A Shared Task on Bandit Learning for Machine Translation

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 514–524 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics A Shared Task on Bandit Learning for Machine Translation Artem Sokolov∗,⋄ and Julia Kreutzer⋄ and Kellen Sunderland∗ and Pavel Danchenko∗ and Witold Szymaniak∗ and Hagen F¨urstenau∗ and Stefan Riezler⋄ ∗Amazon Development Center Germany, Berlin and ⋄Heidelberg University, Germany Abstract We introduce and describe the results of a novel shared task on bandit learning for machine translation. The task was orga- nized jointly by Amazon and Heidelberg University for the first time at the Sec- ond Conference on Machine Translation (WMT 2017). The goal of the task is to encourage research on learning machine translation from weak user feedback in- stead of human references or post-edits. On each of a sequence of rounds, a ma- chine translation system is required to pro- pose a translation for an input, and re- ceives a real-valued estimate of the qual- ity of the proposed translation for learn- ing. This paper describes the shared task’s learning and evaluation setup, using ser- vices hosted on Amazon Web Services (AWS), the data and evaluation metrics, and the results of various machine transla- tion architectures and learning protocols. 1 Introduction Bandit Learning for machine translation (MT) is a framework to train and improve MT systems by learning from weak or partial feedback: In- stead of a gold-standard human-generated trans- lation, the learner only receives feedback to a sin- gle proposed translation (hence the term ‘partial’), in form of a translation quality judgement (a real number which can be as weak as a binary accep- tance/rejection decision). In the shared task, user feedback was simu- lated by a service hosted on Amazon Web Ser- vices (AWS). Participants can submit translations and receive feedback on translation quality. This is used to adapt an out-of-domain MT model, pre-trained on mostly news texts, to a new do- main (e-commerce), for the translation direction of German (DE) to English (EN). While in our setup feedback was simulated by evaluating a re- ward function on the predicted translation against a gold standard reference, the reference translation itself was never revealed to the learner, neither at training nor at test time. This learning scenario has been investigated under the names of learning from bandit feedback1 or reinforcement learning (RL)2, and has important real world applications such as online advertising (Chapelle et al., 2014). In the advertising application, the problem is to select the best advertisement for a user visiting a publisher page. A key element is to estimate the click-through rate (CTR), i.e., the probability that an ad will be clicked by a user so that the adver- tiser has to pay. This probability is modeled by features representing user, page, and ad, and is es- timated by trading off exploration (a new ad needs to be displayed in order to learn its click-through rate) and exploitation (displaying the ad with the current best estimate is better in the short …

Content truncated to 500 words