Bandit structured prediction for learning from partial feedback in statistical machine translation

Bandit Structured Prediction for Learning from Partial Feedback in Statistical Machine Translation Artem Sokolov sokolov@cl.uni-heidelberg.de Stefan Riezler∗ riezler@cl.uni-heidelberg.de Computational Linguistics and IWR∗ Heidelberg University, 69120 Heidelberg, Germany Tanguy Urvoy tanguy.urvoy@orange.com Orange Labs, 2 Avenue Pierre Marzin, 22307 Lannion, France Abstract We present an approach to structured prediction from bandit feedback, called Bandit Structured Prediction, where only the value of a task loss function at a single predicted point, instead of a correct structure, is observed in learning. We present an application to discriminative rerank- ing in Statistical Machine Translation (SMT) where the learning algorithm only has access to a 1 − BLEU loss evaluation of a predicted translation instead of obtaining a gold standard reference translation. In our experiment bandit feedback is obtained by evaluating BLEU on reference translations without revealing them to the algorithm. This can be thought of as a simulation of interactive machine translation where an SMT system is personalized by a user who provides single point feedback to predicted translations. Our experiments show that our approach improves translation quality and is comparable to approaches that employ more in- formative feedback in learning. 1 Introduction Learning from bandit1 feedback describes an online learning scenario, where on each of a se- quence of rounds, a learning algorithm makes a prediction, and receives partial information in terms of feedback to a single predicted point. In difference to the full information supervised scenario, the learner does not know what the correct prediction looks like, nor what would have happened if it had predicted differently. This scenario has (financially) important real world ap- plications such as online advertising (Chapelle et al., 2014) that showcases a tradeoff between exploration (a new ad needs to be displayed in order to learn its click-through rate) and exploita- tion (displaying the ad with the current best estimate is better in the short term). Crucially, in this scenario it is unrealistic to expect more detailed feedback than a user click on the displayed ad. Similar to the online advertising scenario, there are many potential applications of bandit learning to NLP situations where feedback is limited for various reasons. For example, on- line learning has been applied successfully in interactive statistical machine translation (SMT) (Bertoldi et al., 2014; Denkowski et al., 2014; Green et al., 2014). Post-editing feedback clearly is limited by its high cost and by the required expertise of users, however, current approaches force the full information supervised scenario onto the problem of learning from user post-edits. 1The name is inherited from a model where in each round a gambler pulls an arm of a different slot machine (“one-armed bandit”), with the goal of maximizing his reward relative to the maximal possible reward, without apriori knowledge of the optimal slot machine. Proceedings of MT Summit XV, vol.1: MT Researchers' Track Miami, Oct 30 - Nov 3, 2015 | p. 160 Bandit learning would allow to learn from partial user feedback that is easier and faster to ob- tain than full information. An example where user …

Content truncated to 500 words