The UMD Neural Machine Translation Systems at WMT17 Bandit Learning Task

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 667–673 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics The UMD Neural Machine Translation Systems at WMT17 Bandit Learning Task Amr Sharaf and Shi Feng and Khanh Nguyen and Kiant´e Brantley and Hal Daum´e III Department of Computer Science University of Maryland, College Park {amr,shifeng,kxnguyen,kdbrant,hal}@cs.umd.edu Abstract We describe the University of Mary- land machine translation systems submit- ted to the WMT17 German-English Ban- dit Learning Task. The task is to adapt a translation system to a new domain, using only bandit feedback: the system receives a German sentence to translate, produces an English sentence, and only gets a scalar score as feedback. Targeting these two challenges (adaptation and bandit learn- ing), we built a standard neural machine translation system and extended it in two ways: (1) robust reinforcement learning techniques to learn effectively from the bandit feedback, and (2) domain adapta- tion using data selection from a large cor- pus of parallel data. 1 Introduction We describe the University of Maryland systems for bandit machine translation. For the shared translation task of the EMNLP 2017’s second conference on machine translation (WMT17), we focused on the task of bandit machine transla- tion. This shared task was set up, consistent with (Kreutzer et al., 2017), simultaneously as a bandit learning problem and a domain adaptation prob- lem. This raises the natural question: can we com- bine these potentially complementary information sources? To investigate this question, we started from a standard neural machine translation (NMT) setup §21, and then we: 1. applied domain adaptation techniques by data selection (Moore and Lewis, 2010) to the out- of-domain data, with the goals of filtering out 1Our implementation is based on OpenNMT (Klein et al., 2017), an open-source toolkit for neural MT. harmful data and fine-tuning the training pro- cess to focus only on relevant sentences (§4). 2. trained robust reinforcement learning algo- rithms that can effectively learn from bandit feedback (§3); this allows our model to “test” proposed generalizations and adapt from the provided feedback signals. Tackling the problem of learning with ban- dit feedback is important because neural machine translation systems, like other natural language processing technology, currently learn almost ex- clusively from labeled data for a specific domain. While this approach is useful, it cannot scale to a broad variety of language and domains, as linguis- tic systems often cannot generalize well beyond their training data. Machine translation systems need to be able to learn to improve their perfor- mance from naturalistic interaction with users in addition to labeled data. Bandit feedback (Robbins, 1985) offers systems the opportunity to “test” proposed generalizations and receive feedback on their performance; par- ticularly interesting are contextual bandit systems, which make predictions based on a given input context (Auer et al., 2002; Langford and Zhang, 2008; Beygelzimer et al., 2010; Dudik et al., 2011). For example, a neural translation system trained on parliament proceedings often performs quite poorly at translating anything …

Content truncated to 500 words