Machine Translation System Selection from Bandit Feedback

Machine Translation System Selection from Bandit Feedback Jason Naradowsky narad@preferred.jp Preferred Networks, Tokyo, Japan Xuan Zhang xuanzhang@jhu.edu Kevin Duh kevinduh@cs.jhu.edu Johns Hopkins University, Baltimore, USA Abstract Adapting machine translation systems in the real world is a difficult problem. In contrast to offline training, users cannot provide the type of fine-grained feedback (such as correct translations) typically used for improving the system. Moreover, different users have different translation needs, and even a single user’s needs may change over time. In this work we take a different approach, treating the problem of adaptation as one of selection. Instead of adapting a single system, we train many translation systems using different architec- tures, datasets, and optimization methods. Using bandit learning techniques on simulated user feedback, we learn a policy to choose which system to use for a particular translation task. We show that our approach can (1) quickly adapt to address domain changes in translation tasks, (2) outperform the single best system in mixed-domain translation tasks, and (3) make effective instance-specific decisions when using contextual bandit strategies. 1 Introduction Recent advances in machine translation have greatly improved translation quality on in-domain data (Vaswani et al., 2017). But choosing the best system to deploy for a given translation task can be difficult, as many different systems could be considered approximately state-of-the-art, and there is not a single system which is best for all situations. For instance, while neural machine translation (NMT) traditionally excels in big data scenarios, when data is scarce it is not uncommon for a statistical phrased-based translation (SMT) to be the better choice. Even different hyperparameter settings of the same model may yield systems which each excel at different translations tasks. In this work we explore the practical question of how best to deploy and improve an MT service over time. One solution to this problem is adaptation, where the model continues to train in an online manner during deployment, and the model parameters are updated in response to new types of data. However, adapting a system in this manner has the potential to cause catastrophic forgetting (Kirkpatrick et al., 2016): the model parameters shift too much, and performance on the original translation task declines. A second practical concern is that for translation services deployed in the real world, the degree of feedback is often limited. Users of Google Translate can rate the quality of a trans- lation as “helpful” or “wrong”, and Facebook users can use an ordinal scale from 1 to 5, but neither can realistically ask a user to provide a reference translation. Furthermore, the user provides feedback only a single time per translation, and multiple users are unlikely to ask for Proceedings of the 14th Conference of the Association for Machine Translation in the Americas October 6 - 9, 2020, Volume 1: MT Research Track Page 50 Figure 1: Basic setup: Bandit Learning Agent selects the MT system to use at time t. Based on user feedback (e.g. thumbs up/down or other signals), the bandit learning agent …

Content truncated to 500 words