Counterfactual Learning from Bandit Feedback under Deterministic Logging : A Case Study in Statistical Machine Translation

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2566–2576 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Counterfactual Learning from Bandit Feedback under Deterministic Logging: A Case Study in Statistical Machine Translation Carolin Lawrence Heidelberg University, Germany Artem Sokolov Amazon Development Center & Heidelberg University, Germany {lawrence,sokolov,riezler}@cl.uni-heidelberg.de Stefan Riezler Heidelberg University, Germany Abstract The goal of counterfactual learning for statistical machine translation (SMT) is to optimize a target SMT system from logged data that consist of user feed- back to translations that were predicted by another, historic SMT system. challenge arises by the fact that risk-A averse commercial SMT systems deter- ministically log the most probable trans- lation. The lack of sufficient exploration of the SMT output space seemingly con- tradicts the theoretical requirements for counterfactual learning. We show that counterfactual learning from determinis- tic bandit logs is possible nevertheless by smoothing out deterministic components in learning. This can be achieved by ad- ditive and multiplicative control variates that avoid degenerate behavior in empir- ical risk minimization. Our simulation experiments show improvements of up to 2 BLEU points by counterfactual learn- ing from deterministic bandit feedback. 1 Introduction Commercial SMT systems allow to record large amounts of interaction log data at no cost. Such logs typically contain a record of the source, the translation predicted by the system, and the user feedback. The latter can be gathered di- rectly if explicit user quality ratings of transla- tions are supported, or inferred indirectly from the interaction of the user with the translated content. Indirect feedback in form user clicks on displayed ads has been shown to be a valu- able feedback signal in response prediction for display advertising (Bottou et al., 2013). Similar to the computational advertising scenario, one could imagine a scenario where SMT systems are optimized from partial information in form of user feedback to predicted translations, instead of from manually created reference translations. This learning scenario has been investigated in the areas of bandit learning (Bubeck and Cesa- Bianchi, 2012) or reinforcement learning (RL) (Sutton and Barto, 1998). Figure 1 illustrates the learning protocol using the terminology of ban- dit structured prediction (Sokolov et al., 2016; Kreutzer et al., 2017), where at each round, a system (corresponding to a policy in RL terms) makes a prediction (also called action in RL, or pulling an arm of a bandit), and receives a re- ward, which is used to update the system. Figure 1: Online learning from partial feedback. Counterfactual learning attempts to reuse ex- isting interaction data where the predictions have been made by a historic system different from the target system. This enables offline or batch learning from logged data, and is important if 2566 online experiments that deploy the target system are risky and/or expensive. Counterfactual learn- ing tasks include policy evaluation, i.e. estimat- ing how a target policy would have performed if it had been in control of choosing the predictions for which the rewards were logged, and policy …

Content truncated to 500 words